Artificial intelligence-based video revisit method and device, computer device and medium

Abstract

The application relates to the technical field of artificial intelligence, in particular to a video revisit method and device based on artificial intelligence, computer equipment and a medium. The method extracts reference features of a reference image, initializes the iteration number as zero, maps the iteration number to a collection time point, collects real-time images at the collection time point and extracts real-time features, inputs the real-time features into a detection model when the similarity of the reference features and the real-time features is greater than a similarity threshold value, outputs a face detection result, sends a revisit question to a target user when the number of the face detection result is the same as a number reference value, receives a revisit answer, increases the iteration number by one, returns to execute an iteration step, obtains a revisit question and a revisit answer with a corresponding relationship, and adaptively selects the termination or performance of the revisit according to a judgment condition, so that the revisit cost is reduced, the revisit efficiency is effectively improved, and the user experience of a customer for a financial service platform is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a video follow-up method and device based on artificial intelligence, a computer device and a medium. BACKGROUND

[0002] With the rise of artificial intelligence technology, video follow-up technology based on artificial intelligence models has been widely used in financial service platforms, which can be insurance systems, bank systems, transaction systems, order systems, etc. The financial service platform can support shopping, socializing, interactive games, resource transfer, etc., and can also have functions such as applying for loans, credit cards, or purchasing insurance, financial products, etc.

[0003] In the user follow-up scene of the financial service platform, the video follow-up technology applying intelligent questioning and answering can effectively reduce the human cost of follow-up and improve the efficiency of follow-up.

[0004] However, the existing online follow-up method still needs human participation and consumes a large amount of human resources. One improvement is to combine artificial intelligence models for intelligent questioning and answering to realize online follow-up. However, the artificial intelligence model cannot guarantee that the follow-up process is specific to a particular user, which may result in ineffective questioning and answering and low efficiency of online follow-up. Therefore, how to improve the follow-up efficiency for a specific user has become a problem to be solved. SUMMARY

[0005] Therefore, the embodiments of the present application provide a video follow-up method and device based on artificial intelligence, a computer device and a medium to solve the problem of low efficiency of online follow-up for a specific user.

[0006] In a first aspect, the embodiments of the present application provide a video follow-up method based on artificial intelligence, which comprises:

[0007] obtaining a reference image of a target user, extracting image features of the reference image to obtain reference features;

[0008] initializing the number of iterations to zero, mapping the number of iterations to a collection time point, collecting a real-time image of the target user at the collection time point, extracting real-time features of the real-time image, and calculating the similarity between the reference features and the real-time features;

[0009] when the similarity is greater than a preset similarity threshold, inputting the real-time features into a trained detection model for face detection, and outputting at least one face detection result;

[0010] count a number of all face detection results, and when the number is equal to a preset number reference value, send a revisit question corresponding to a question serial number matched with the iteration number to the target user, and receive a revisit answer corresponding to the revisit question;

[0011] increase the iteration number by one, return to execute the step of mapping the iteration number to a collection time point, and stop iteration until all revisit questions have been sent to the target user, and obtain at least one set of revisit questions and revisit answers having a corresponding relationship.

[0012] In a second aspect, an embodiment of the present application provides a video revisit device based on artificial intelligence, which comprises:

[0013] a feature extraction module configured to obtain a reference image of a target user, extract image features of the reference image, and obtain reference features;

[0014] a similarity calculation module configured to initialize an iteration number as zero, map the iteration number to a collection time point, collect a real-time image of the target user at the collection time point, extract real-time features of the real-time image, and calculate a similarity between the reference features and the real-time features;

[0015] a face detection module configured to, when the similarity is greater than a preset similarity threshold, input the real-time features into a trained detection model for face detection, and output at least one face detection result;

[0016] a question sending module configured to count a number of all face detection results, and when the number is equal to a preset number reference value, send a revisit question corresponding to a question serial number matched with the iteration number to the target user, and receive a revisit answer corresponding to the revisit question;

[0017] an iteration termination module configured to increase the iteration number by one, return to execute the step of mapping the iteration number to a collection time point, and stop iteration until all revisit questions have been sent to the target user, and obtain at least one set of revisit questions and revisit answers having a corresponding relationship.

[0018] In a third aspect, an embodiment of the present application provides a computer device, which comprises a processor, a memory, and a computer program stored in the memory and executable on the processor, and the processor implements the video revisit method of the first aspect when executing the computer program.

[0019] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executable on a processor to implement the video revisit method of the first aspect.

[0020] The beneficial effects of the embodiments of the present application compared with the prior art are:

[0021] The reference image of the target user is acquired, the image features of the reference image are extracted, the reference features are obtained, the iteration number is initialized to zero, the iteration number is mapped to the acquisition time point, the real-time image of the target user is acquired at the acquisition time point, the real-time features of the real-time image are extracted, the similarity of the reference features and the real-time features is calculated, when the similarity is greater than a preset similarity threshold, the real-time features are input into the trained detection model for face detection, at least one face detection result is output, the number of all face detection results is counted, when the number is the same as a preset number reference value, the follow-up question corresponding to the problem serial number matched with the iteration number is sent to the target user, the follow-up answer to the corresponding follow-up question is received, the iteration number is increased by one, the step of mapping the iteration number to the acquisition time point is returned to execute, until all follow-up questions have been sent to the target user, the iteration is stopped, at least one set of follow-up questions and follow-up answers with corresponding relationship are obtained, through multiple threshold value judgments, it can be ensured that the follow-up process is for the target user, and the termination or continuation of follow-up can be adaptively selected according to the judgment situation, while reducing the follow-up cost, the follow-up efficiency can be effectively improved, and the user experience of customers for the financial service platform can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0023] Figure 1 is an application environment schematic diagram of a video follow-up method based on artificial intelligence provided by the first embodiment of the present application;

[0024] Figure 2 is a flowchart of a video follow-up method based on artificial intelligence provided by the first embodiment of the present application;

[0025] Figure 3 is a structural schematic diagram of a video follow-up device based on artificial intelligence provided by the second embodiment of the present application;

[0026] Figure 4 is a structural schematic diagram of a computer device provided by the third embodiment of the present application. DETAILED DESCRIPTION

[0027] In the following description, for purposes of explanation and not limitation, specific details are set forth such as particular architectures, techniques, etc. in order to provide a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application can be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known methods, devices, circuits, and

[0028] It is to be understood that the terminology "includes", "has", "holds", "contains" or "comprises", "comprising", or "including" when used in the present specification and in the following claims specifies the presence of the stated features, integers, actions, operations, elements, or components, but do not preclude the presence or addition of one or more other features, integers, actions, operations, elements, components, or groups thereof.

[0029] It is also to be understood that the terminology "and / or" when used in the present specification and in the following claims, refers to at least one of the items, or any combination of one or more of the items, associated with the "and / or" term.

[0030] As used in the present specification and in the following claims, the term "if" can be interpreted as meaning "when" or "once" or "in response to a determination" or "in response to a detection" depending on the context. Similarly, the phrase "if determined" or "if detected [the described condition or event]" can be interpreted as meaning "once determined" or "in response to a determination" or "once detected [the described condition or event]" or "in response to a detection [the described condition or event]" depending on the context.

[0031] In addition, the terms "first", "second", "third", etc. in the description of the present specification and in the following claims are only used to distinguish descriptions, and cannot be understood as indicating or implying relative importance.

[0032] Reference in the specification to "one embodiment" or "some embodiments" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the application. The appearances of the phrases "in one embodiment", "in some embodiments", "in other embodiments", "in additional embodiments", etc. in various places in the specification are not necessarily all referring to the same embodiment, although they can. The terms "comprise", "comprising", "have", "having", "include", "including", and "contain", "containing" are used in the specification to mean "including but not limited to", unless otherwise indicated.

[0033] The embodiment of the present application can acquire and process related data based on artificial intelligence technology. Among them, artificial intelligence (AI) is to use digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0034] The basic technology of artificial intelligence generally includes technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. The artificial intelligence software technology mainly includes computer vision technology, robot technology, biometric technology, speech processing technology, natural language processing technology, and machine learning / deep learning, etc.

[0035] It should be understood that the size of the serial number of each step in the following embodiment does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

[0036] In order to illustrate the technical scheme of the present application, the following will be explained by specific embodiments.

[0037] The video revisit method based on artificial intelligence provided by the embodiment of the present application can be applied in application environments such as Figure 1 , wherein the user end and the service end communicate. The user end includes but is not limited to palmtop computer, desktop computer, notebook computer, ultra-mobile personal computer (UMPC), netbook, cloud terminal device, personal digital assistant (PDA) and other computer devices. The service end can be an independent server, or a cloud server providing cloud services, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, content distribution network (CDN), and basic cloud computing services such as big data and artificial intelligence platform.

[0038] The client and the service end can be deployed in a financial service platform, which can be an insurance system, a bank system, a transaction system, an order system, etc. The financial service platform can support functions such as shopping, socializing, interactive games, and resource transfer, and can also have functions such as applying for loans, credit cards, or purchasing insurance, financial products, etc. The service end can perform video revisit tasks, so that the financial customer revisit maintenance in the financial service platform is carried out at a lower cost, and the user experience of the customer for the financial service platform is improved.

[0039] Referring to Figure 2 , Fig. 1 is a flow diagram of a video revisit method based on artificial intelligence provided by an embodiment of the present application. The video revisit method based on artificial intelligence can be applied to Figure 1 a server. The server corresponds to a computer device connected with a user terminal, obtains a reference image provided by a user in the user terminal, sends a revisit question to the user terminal in a revisit process to show the user, and receives a revisit answer provided by the user in the user terminal. The computer device corresponding to the server is internally deployed with a trained detection model which can be used for face detection on an input image. As shown in Figure 2 , the video revisit method can include the following steps:

[0040] Step S201, obtaining a reference image of a target user, extracting image features of the reference image to obtain reference features.

[0041] The target user can be an object user in a financial service platform who needs to perform video revisit, for example, the target user can be a user who purchases a financial product or a user who consults financial information, and the reference image can be used to represent the basic face information of the target user. The reference features can be used to represent the face image features of the target user in the reference image.

[0042] Specifically, the server sends a video revisit request for the target user to the user terminal. The target user can receive the video revisit request through the user terminal. After receiving the video revisit request in the user terminal, the face image of the target user is collected by using the camera configured by the user terminal. If the collected face image meets the preset condition, the collected face image is taken as the reference image. The preset condition can be used to determine whether the collected face image can represent the face information of the target user. The preset condition can be a clarity condition, a brightness condition, etc. For example, when the preset condition is the brightness condition, the collected face image is converted into the HSV space to obtain the brightness value corresponding to each pixel point in the collected face image. The average value of the brightness values corresponding to all pixel points is compared with the preset brightness threshold. If the average value is greater than the zero threshold, it means that the brightness condition is met, i.e., the preset condition is met.

[0043] The image feature extraction of the reference image can be realized by using a trained encoder. The trained encoder can be used to extract the image features of the input reference image. The trained encoder can use the encoder part of the trained face positioning model or the encoder part of the trained face recognition model.

[0044] Optionally, obtaining the reference image of the target user includes:

[0045] Searching for a registered image corresponding to the target user from a preset database.

[0046] The snapshot image of the target user is obtained at a preset time point, similarity calculation is performed on the snapshot image and the registration image, and when the similarity calculation result is greater than a preset reference threshold, the snapshot image is determined as the reference image.

[0047] The database includes N groups of users and registration images that have a corresponding relationship, N is an integer greater than zero, the registration image can be image information provided by the user when registering for a corresponding service, one registration image corresponds to one user, and there is a registration image corresponding to the target user in the database.

[0048] The preset time point can be a time point for taking a snapshot of the target user, and the snapshot image can be an image of the target user captured by the camera at the preset time point.

[0049] The reference threshold can be used to determine whether the snapshot image and the registration image are similar enough, if the similarity calculation result is greater than the preset reference threshold, it means that the snapshot image and the registration image are similar enough, and the snapshot image can be used as the reference image of the target user, if the similarity calculation result is less than or equal to the preset reference threshold, it means that the snapshot image and the registration image are not similar enough, and the snapshot image cannot be used as the reference image of the target user.

[0050] Specifically, the preset time point can be set to multiple, and the multiple preset time points can conform to a preset sampling frequency, for example, the sampling frequency can be set to 30 frames per minute, that is, a snapshot image is obtained every two seconds, after obtaining multiple snapshot images, similarity calculation is performed on each snapshot image and the registration image respectively, to obtain the similarity calculation result corresponding to the snapshot image, the maximum value of all similarity calculation results is determined, and the maximum value is compared with the preset reference threshold, if the maximum value is greater than the preset reference threshold, the snapshot image corresponding to the maximum value is used as the reference image, in this embodiment, the preset reference threshold can be set to 0.7, and the implementer can adjust the reference threshold according to the actual situation.

[0051] The similarity can be calculated by Euclidean distance, cosine similarity, etc., in this embodiment, the similarity calculation is directly performed on the snapshot image and the registration image, it should be noted that the image size of the snapshot image and the registration image should be consistent.

[0052] In one embodiment, the same encoder can be used to extract the image features of the snapshot image and the registration image, and the similarity between the extracted image features is calculated, at this time, the similarity between the extracted image features can still be calculated by Euclidean distance, cosine similarity, etc., and the similarity between the extracted image features is used as the similarity of the snapshot image and the registration image.

[0053] In this embodiment, according to the similarity between the snapshot image and the registered image, the snapshot image is determined as the reference image, which can ensure that the information contained in the snapshot image is the face information of the target user, and at the same time can improve the accuracy of the reference image representing the face information of the target user. Since the registered image of the user is a historically stored image, it may be difficult to accurately represent the real-time face information of the user, for example, the current hairstyle, dress, etc. of the user may be inconsistent with the information in the registered image. However, by using the snapshot image as the reference image, a more accurate reference image can be obtained for subsequent use, avoiding misjudgment caused by changeable information such as hairstyle and dress, thereby improving the accuracy of the video revisit process.

[0054] The above steps of obtaining the reference image of the target user, extracting the image features of the reference image, and obtaining the reference features provide a reference for real-time personnel verification in the subsequent revisit process, thereby ensuring that the video revisit process is for the target user and improving the pertinence of the video revisit process.

[0055] In step S202, the iteration number is initialized to zero, the iteration number is mapped to the acquisition time point, the real-time image of the target user is acquired at the acquisition time point, the real-time features of the real-time image are extracted, and the similarity between the reference features and the real-time features is calculated.

[0056] The iteration number can represent the iteration round of each iteration process, the acquisition time point can be used to determine the acquisition time of the target user image, the real-time image can refer to the target user image acquired at the acquisition time point, and the real-time features can be used to represent the image features of the real-time image.

[0057] Specifically, the mapping of the iteration number to the acquisition time point can be realized by a mapping function, and the mapping function can include the mapping relationship between the iteration number and the acquisition time point. The extraction of the real-time features can also use the above trained encoder.

[0058] In this embodiment, the similarity between the reference features and the real-time features can be represented by the cosine similarity, and the value range of the cosine similarity is [0, 1]. The closer the similarity is to 1, the more similar the reference features and the real-time features used to calculate the similarity are. The closer the similarity is to 0, the less similar the reference features and the real-time features used to calculate the similarity are.

[0059] Optionally, the mapping of the iteration number to the acquisition time point includes:

[0060] An acquisition time point is obtained, and the acquisition time point is used to represent the time when the revisit answer is received in the previous iteration process.

[0061] The iteration number is multiplied by a preset coefficient, the sum of the multiplication result and the acquisition time point is calculated, and the calculation result is determined as the acquisition time point.

[0062] The question and answer termination time point can be a time when the question and answer session ends in the previous iteration process, and in this embodiment, the time when the feedback answer is received in the previous iteration process is taken as the question and answer termination time point. The preset coefficient can be used to control the time interval between the question and answer termination time point and the collection time point according to the iteration number, and in this embodiment, the preset coefficient can be set to 1.

[0063] In an embodiment, the termination time point of the previous iteration process can also be taken as the question and answer termination time point, or the updating time point of the iteration number can be taken as the question and answer termination time point.

[0064] In an embodiment, the starting time point of the entire iteration process can be obtained, the multiplication result of the iteration number multiplied by the preset coefficient is added to the starting time point to obtain a reference time point, and the question and answer termination time point is compared with the reference time point. If the question and answer termination time point is less than the reference time point, the reference time point is taken as the collection time point, and if the question and answer termination time point is greater than or equal to the reference time point, the question and answer termination time point is taken as the collection time point, so as to provide sufficient waiting time between the answer sessions and avoid too close problem sending to cause poor user experience.

[0065] In this embodiment, the collection time point is obtained according to the iteration number, so as to provide sufficient waiting time between the answer sessions, avoid too close problem sending to cause poor user experience, and improve the user experience of the target user in the video feedback process.

[0066] Optionally, after calculating the similarity between the reference feature and the real-time feature, the method further comprises:

[0067] When the similarity is less than or equal to a preset similarity threshold, the first abnormal statistical value is increased by one, and the initial value of the first abnormal statistical value is zero;

[0068] Correspondingly, the iteration number is increased by one, and the step of mapping the iteration number to the collection time point is executed again until all feedback questions are sent to the target user, and the iteration is stopped, so as to obtain at least one group of feedback questions and feedback answers having a corresponding relationship.

[0069] The iteration number is increased by one, and the step of mapping the iteration number to the collection time point is executed again until all feedback questions are sent to the target user or the ratio of the first abnormal statistical value to the iteration number is less than a preset reference ratio, and the iteration is stopped, so as to obtain at least one group of feedback questions and feedback answers having a corresponding relationship.

[0070] The first abnormal statistical value can be used to represent the total number of frames in which the target user is not detected, and the preset reference ratio can be used to determine whether the target user has an abnormal situation, and the abnormal situation can include user change, target user leaving, etc.

[0071] Specifically, the preset similarity threshold can be set to 0.6, and the implementer can adjust the preset similarity threshold according to the actual situation. When the similarity is less than or equal to the preset similarity threshold, it indicates that the target user may leave the camera coverage range or the target user and other users change at this time, and the first abnormal statistical value is increased by one.

[0072] The preset reference ratio can be set to 0.3, that is, in the process of ten iterations, more than three abnormal situations occur, and the iteration is stopped to ensure that the video revisit is for the target user. It needs to be noted that the embodiment defaults that the target user is in a normal state when the video revisit starts. The judgment of the preset reference ratio is mainly applied to the judgment of the continuous abnormal situation in the video revisit process.

[0073] In the embodiment, the accumulation of the first abnormal statistical value ensures that the video revisit is for the target user in the video revisit process, avoids the situation that the user changes or leaves the camera range in the revisit process, which may lead to the change of the revisit object, thereby improving the pertinence of the video revisit for the target user and avoiding the leakage of the revisit information.

[0074] The above initialization iteration number is zero, the iteration number is mapped to the collection time point, the real-time image of the target user is collected at the collection time point, the real-time feature of the real-time image is extracted, and the similarity of the reference feature and the real-time feature is calculated. The steps of comparing the real-time feature of the real-time image with the reference feature provide a preliminary screening function for subsequent face detection tasks, thereby reducing the calculation amount of the subsequent face detection task and improving the efficiency of the video revisit process.

[0075] Step S203, when the similarity is greater than the preset similarity threshold, inputting the real-time feature into the trained detection model for face detection, and outputting at least one face detection result.

[0076] Among them, the input of the trained detection model can be the real-time feature, the output of the trained detection model can be the face detection result, the trained detection model can adopt a target detection model such as RCNN model, YOLO model, and the face detection result can include a face prediction bounding box and a face category.

[0077] Specifically, when the similarity is greater than the preset similarity threshold, it indicates that the real-time feature can effectively represent the information of the reference feature, that is, the real-time feature contains the corresponding face feature information in the reference feature. At this time, the real-time feature is input into the trained detection model for face detection, which can avoid the real-time feature corresponding to the real-time image irrelevant to the face information of the target user from being processed by face detection, causing calculation redundancy.

[0078] In the embodiment, since the task of the trained detection model is face detection, it can be considered that the prediction category of the trained detection model is only one, i.e., the face category, and thus it can be considered that the output of the trained detection model is at least one face bounding box belonging to the face category. The face bounding box can be represented by a bounding box top-left corner point and a bounding box bottom-right corner point, and a rectangular frame can be determined based on the bounding box top-left corner point and the bounding box bottom-right corner point, which is the face bounding box.

[0079] The trained detection model can detect all face images contained in the real-time image, not limited to the face image of the target user, and thus the trained detection model outputs at least one face detection result.

[0080] The above step of inputting the real-time feature into the trained detection model for face detection when the similarity is greater than the preset similarity threshold and outputting at least one face detection result can effectively avoid redundant reasoning of the trained detection model and waste of computing resources, so as to perform face detection only on the real-time image containing the face information of the target user, improve the computing efficiency, and improve the video revisit efficiency for the target user.

[0081] In step S204, the number of all face detection results is counted, and when the number is the same as a preset number reference value, a revisit question corresponding to the problem sequence number matching the iteration number is sent to the target user, and a revisit answer to the revisit question is received.

[0082] The number can refer to the number of face detection results output by the trained detection model in the current iteration process, the preset number reference value can be used to determine whether the real-time image contains only the face information of the target user, the problem sequence number can refer to the identification of the revisit question, the revisit question can refer to a question that needs to be answered by the target user in the video revisit process, and the revisit answer can refer to the answer result of the target user to the revisit question.

[0083] Specifically, in the embodiment, the preset number reference value can be set to 1, and the implementer can adjust the preset number reference value according to the actual situation. For example, if the video revisit process allows one other user to listen, the preset number reference value can be adjusted to 2.

[0084] Generally, the follow-up process includes several follow-up questions, and all follow-up questions usually conform to the order of the questions, therefore, when the number is the same as the preset number reference value, the follow-up question corresponding to the question number matching the iteration number is sent to the target user, for example, when the iteration number is 1, the first follow-up question is sent to the user, and the first follow-up question can be an identity confirmation type, a polite inquiry type, etc., the identity confirmation type can be “Are you XX?”, the polite inquiry type can be “Are you available now?” and the like, in this embodiment, the follow-up question can be presented in the form of text on the user end for the target user to view and answer.

[0085] In an embodiment, the text corresponding to the follow-up question can also be converted into a question voice through a trained voice conversion model, and then the question voice is played through the user end, it should be noted that the follow-up question in the form of text and voice can be combined for display.

[0086] Optionally, after counting the number of all face detection results, the following is further included:

[0087] When the number is not the same as the preset number reference value, the second abnormality statistical value is increased by one, and the initial value of the second abnormality statistical value is zero;

[0088] Correspondingly, the iteration number is increased by one, and the step of mapping the iteration number to the collection time point is executed again, until all follow-up questions have been sent to the target user, the iteration is stopped, and at least one set of follow-up questions and follow-up answers with a corresponding relationship is obtained, including:

[0089] The iteration number is increased by one, and the step of mapping the iteration number to the collection time point is executed again, until all follow-up questions have been sent to the target user or the second abnormality statistical value is the same as the preset abnormality reference value, the iteration is stopped, and at least one set of follow-up questions and follow-up answers with a corresponding relationship is obtained.

[0090] The second abnormality statistical value can be used to represent the total number of frames in which the number of people in the camera does not meet the expected number, and the preset abnormality reference value can be used to determine whether the number of people is abnormal, and the abnormality can include too many users, too few users, etc.

[0091] Specifically, the preset abnormality reference value can be set to 5, and the implementer can adjust the preset abnormality reference value according to the actual situation, when the number is not the same as the preset number reference value, it means that there is an abnormal situation of the number of people, and the second abnormality statistical value is increased by one.

[0092] In this embodiment, the second abnormal statistical value is accumulated, so as to ensure that the video follow-up process is for the target user, avoid other users eavesdropping in the follow-up process, and prevent the follow-up information from being leaked, thereby improving the pertinence of the video follow-up for the target user, that is, improving the privacy of the follow-up information.

[0093] Optionally, the follow-up answer to the follow-up question comprises:

[0094] determining a questioning time point at which the follow-up question is sent to the target user, starting from the questioning time point, receiving the voice data of the target user frame by frame until the difference between the current frame time point and the questioning time point is greater than a preset difference threshold, stopping receiving, and obtaining all the voice data;

[0095] After splicing all the voice data, the splicing result is input into the trained voice recognition model, and the follow-up answer to the follow-up question is output.

[0096] The questioning time point can represent the time point at which the follow-up question is sent to the target user, the implementer can take the time point at which the user end starts to display the follow-up question as the questioning time point, the voice data can be the voice information of the target user, the current frame time point can be the time corresponding to the current frame, the preset difference threshold can be used to determine whether the answering time of the target user has reached the preset time length, and the trained voice recognition model can convert the input voice data into text data.

[0097] Specifically, the voice data can be received through a microphone deployed by the user end. In this embodiment, the receiving frequency of the voice data can be set to 30 frames per second, and the preset difference threshold can be set to 300 frames, that is, the voice data within ten seconds is received starting from the questioning time point, and at this time, ten seconds is the preset time length. The implementer can adjust the receiving frequency of the voice data and the preset difference threshold according to the actual situation.

[0098] Since the number of received frames is fixed, the amount of voice data is fixed, and accordingly, after splicing all the voice data, the size of the splicing result is fixed, which can be directly input into the trained voice recognition model for text conversion.

[0099] In an implementation, while receiving the voice data, the camera can cooperatively collect real-time images frame by frame, and perform lip shape detection and occlusion detection on the real-time images. The lip shape detection can be implemented by using a trained action unit recognition model, and the occlusion detection can be implemented by using a trained occlusion detection model. Common occlusions can include masks, palms, arms, etc. For any frame, the voice data received by the frame can correspond to one of a human voice audio category and a noise category. Combined with the lip shape detection result and the occlusion detection result of the frame, the answer authenticity of the target user can be further verified. For example, if the voice data received by the frame corresponds to the human voice audio category, but the lip shape detection result of the frame is not a sound-making lip shape, or the lip shape does not conform to the sound-making lip shape corresponding to the voice data, there can be a situation of answering on behalf of others, and a warning information can be sent to the server.

[0100] In the embodiment, the audio information of the answer of the target user can be effectively obtained by collecting voice data frame by frame within a preset time length, the amount of invalid noise collected is reduced, the calculation amount of the voice conversion model is reduced, and the efficiency of the video revisit of the target user is improved.

[0101] Optionally, starting from the questioning time point, voice data of the target user is received frame by frame until the difference between the current frame time point and the questioning time point is greater than a preset difference threshold, the receiving is stopped, and all voice data is obtained, including:

[0102] Starting from the questioning time point, voice data of the target user is received frame by frame, and when K consecutive frames of voice data are noise data, it is determined that a preset condition is met.

[0103] When the preset condition is met or the difference between the current frame time point and the questioning time point is greater than the preset difference threshold, the receiving is stopped, and all voice data is obtained.

[0104] The preset condition can be used to determine whether the target user has finished answering, and the noise data can be voice data that does not contain human voice audio information.

[0105] Specifically, in the embodiment, K can be set to 90 frames, and the implementer can adjust the value of K according to the actual situation, but the value of K should be less than the above-mentioned preset difference threshold.

[0106] When the preset condition is met, it means that the target user has not made a sound for a long time, and it is considered that the target user refuses to answer or has finished answering. The voice data is stopped from being continuously received, thereby effectively reducing the answer waiting time of the target user. It should be noted that if the receiving is stopped due to the preset condition, size normalization needs to be performed after all voice data is spliced. The size normalization can be implemented by adding white noise frames to the spliced result. The target of the size normalization is to have the same number of frames as the above-mentioned preset time length.

[0107] In this embodiment, if no human voice data is received for a plurality of continuous frames, the receiving process is stopped, and it is considered that the target user has finished answering, thereby effectively reducing the waiting time of the target user after finishing answering, and improving the user experience of the target user in video follow-up.

[0108] The number of all face detection results is counted, and when the number is the same as the preset number reference value, the follow-up question corresponding to the question number matching the iteration number is sent to the target user, and the step of receiving the follow-up answer to the follow-up question is performed once in a single iteration process, that is, a follow-up answer is completed, without the need for manual participation, thereby improving the efficiency of video follow-up.

[0109] In step S205, the iteration number is increased by one, and the step of mapping the iteration number to the collection time point is returned to be executed until all follow-up questions have been sent to the target user, the iteration is stopped, and at least one set of follow-up questions and follow-up answers with a corresponding relationship is obtained.

[0110] In each iteration process, a set of follow-up questions and follow-up answers can be collected, and when all follow-up questions have been sent to the target user, it means that the follow-up task is completed, and therefore the iteration is stopped, and at least one set of follow-up questions and follow-up answers with a corresponding relationship obtained is recorded.

[0111] Specifically, the implementer can pre-set a question and answer logic, determine the follow-up question matching the iteration number in the current iteration process according to the follow-up answer of the user in the previous iteration process, thereby improving the intelligence of the follow-up process. For example, the follow-up question in the previous iteration process is “whether Y product is purchased”, if the follow-up answer of the user is “yes”, the follow-up question matching the iteration number in the current iteration process can be “how is the experience?”, if the follow-up answer of the user is “no”, the follow-up question matching the iteration number in the current iteration process can be “whether you want to know?” and the like. Such question and answer logic can be directly stored in the server, and the server determines the follow-up question matching the iteration number in each iteration process according to the actual question and answer result.

[0112] The step of increasing the iteration number by one and returning to execute the step of mapping the iteration number to the collection time point until all follow-up questions have been sent to the target user, stopping the iteration, and obtaining at least one set of follow-up questions and follow-up answers with a corresponding relationship, through the iteration mode, the follow-up question asking time is flexibly controlled, without the need for manual participation, so that the target user can complete the follow-up process at home, greatly reducing the cost of the follow-up task, and improving the video follow-up efficiency of the target user.

[0113] In this embodiment, through multiple threshold judgments, it can be effectively ensured that the follow-up process is for the target user, and the termination or continuation of follow-up can be adaptively selected in combination with the judgment situation, thereby reducing the follow-up cost while effectively improving the follow-up efficiency and improving the user experience of the customer for the financial service platform.

[0114] The AI-based video follow-up method according to the above embodiment, Figure 3 The structural block diagram of the AI-based video follow-up device provided by the second embodiment of the present application is shown, and the video follow-up device is applied to a service end. The computer device corresponding to the service end is connected with a user end, obtains a reference image provided by a user in the user end, sends a follow-up question to the user end in a follow-up process to show the user, and receives a follow-up answer provided by the user in the user end. The computer device corresponding to the service end is internally deployed with a trained detection model, which can be used for face detection on an input image. For the convenience of description, only the parts related to the embodiments of the present application are shown.

[0115] Referring to Figure 3 The video follow-up device comprises:

[0116] The feature extraction module 31 is configured to obtain the reference image of the target user, extract the image features of the reference image, and obtain the reference features.

[0117] The similarity calculation module 32 is configured to initialize the iteration number as zero, map the iteration number to a collection time point, collect the real-time image of the target user at the collection time point, extract the real-time features of the real-time image, and calculate the similarity between the reference features and the real-time features.

[0118] The face detection module 33 is configured to input the real-time features into the trained detection model for face detection when the similarity is greater than a preset similarity threshold, and output at least one face detection result.

[0119] The question sending module 34 is configured to count the number of all face detection results, send the follow-up question corresponding to the question serial number matched with the iteration number to the target user when the number is the same as a preset number reference value, and receive the follow-up answer to the follow-up question.

[0120] The iteration termination module 35 is configured to increase the iteration number by one, return to execute the step of mapping the iteration number to the collection time point, and stop iteration when all follow-up questions have been sent to the target user, to obtain at least one set of follow-up questions and follow-up answers with a corresponding relationship.

[0121] Optionally, the feature extraction module 31 comprises:

[0122] An image searching unit is configured to search for a registered image corresponding to the target user from a preset database, the database comprising N sets of users and registered images having a corresponding relationship, N being an integer greater than zero;

[0123] An image determining unit is configured to acquire a snapshot image of the target user at a preset time point, perform similarity calculation on the snapshot image and the registered image, and determine the snapshot image as a reference image when the similarity calculation result is greater than a preset reference threshold.

[0124] Optionally, the similarity calculation module 32 comprises:

[0125] A time point acquiring unit is configured to acquire a question and answer termination time point, the question and answer termination time point being used to represent a time when a revisit answer is received in a previous iteration process;

[0126] A time point determining unit is configured to multiply the iteration number by a preset coefficient, calculate a sum of the multiplication result and the question and answer termination time point, and determine the calculation result as the collection time point.

[0127] Optionally, the video revisit device further comprises:

[0128] A first anomaly statistical module is configured to increase a first anomaly statistical value by one when the similarity is less than or equal to a preset similarity threshold, the first anomaly statistical value having an initial value of zero;

[0129] Correspondingly, the iteration termination module 35 comprises:

[0130] A first iteration termination unit is configured to increase the iteration number by one, return to execute the step of mapping the iteration number to the collection time point, and stop iteration when all revisit questions have been sent to the target user or a ratio of the first anomaly statistical value to the iteration number is less than a preset reference ratio, to obtain at least one set of revisit questions and revisit answers having a corresponding relationship.

[0131] Optionally, the question sending module 34 further comprises:

[0132] A second anomaly statistical unit is configured to increase a second anomaly statistical value by one when the number is different from a preset number reference value, the second anomaly statistical value having an initial value of zero;

[0133] Correspondingly, the iteration termination module 35 comprises:

[0134] A second iteration termination unit is configured to increase the iteration number by one, return to execute the step of mapping the iteration number to the collection time point, and stop iteration when all revisit questions have been sent to the target user or the second anomaly statistical value is equal to the preset anomaly reference value, to obtain at least one set of revisit questions and revisit answers having a corresponding relationship.

[0135] Optionally, the above-mentioned question sending module 34 includes:

[0136] The voice receiving unit is used to determine the questioning time point when the follow-up question is sent to the target user. Starting from the questioning time point, it receives the target user's voice data frame by frame until the difference between the current frame time point and the questioning time point is greater than a preset difference threshold, at which point it stops receiving and obtains all voice data.

[0137] The speech recognition unit is used to concatenate all speech data, input the concatenation result into the trained speech recognition model, and output the corresponding answer to the follow-up question.

[0138] Optionally, the aforementioned voice receiving unit includes:

[0139] The condition judgment subunit is used to receive the target user's voice data frame by frame starting from the questioning time point, and determine that the preset conditions are met when K consecutive frames of voice data are noise data.

[0140] The threshold comparison subunit is used to stop receiving and obtain all voice data when a preset condition is met or the difference between the current frame time point and the question time point is greater than a preset difference threshold.

[0141] It should be noted that the information interaction and execution process between the above modules, units, and sub-units are based on the same concept as the method embodiments of the present invention. For details on their specific functions and technical effects, please refer to the method embodiments section, which will not be repeated here.

[0142] Figure 4 This is a schematic diagram of the structure of a computer device provided in Embodiment 3 of the present invention. Figure 4 As shown, the computer device of this embodiment includes: at least one processor ( Figure 4 Only one is shown in the diagram), a memory, and a computer program stored in the memory and executable on at least one processor, which, when executed by the processor, implements the steps in any of the above-described video replay method embodiments.

[0143] This computer device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that... Figure 4 The examples of computer devices are merely examples and do not constitute a limitation on computer devices. Computer devices may include more or fewer components than shown, or combinations of certain components, or different components, such as network interfaces, displays, and input devices.

[0144] The processor can be a CPU, and can also be other general-purpose processors, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0145] The memory includes a readable storage medium, an internal memory, etc., where the internal memory can be a memory of the computer device, and the internal memory provides an environment for running the operating system and the computer-readable instructions in the readable storage medium. The readable storage medium can be a hard disk of the computer device, and in other embodiments, can also be an external storage device of the computer device, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. Further, the memory can include both an internal storage unit of the computer device and an external storage device. The memory is used to store an operating system, an application program, a BootLoader, data, and other programs, such as program codes of computer programs, etc. The memory can also be used to temporarily store data that has been output or will be output.

[0146] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of each functional unit and module is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of software functional unit. In addition, the specific names of each functional unit and module are only for easy distinction, and do not limit the protection scope of the present application. The specific working process of the unit and module in the above device can refer to the corresponding process in the foregoing method embodiment, which will not be repeated here. If the integrated unit is realized in the form of software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the present application realizes all or part of the processes in the above-mentioned embodiment methods, which can be realized by a computer program to instruct related hardware to complete, and the computer program can be stored in a computer readable storage medium. The computer program includes computer program code, which can be in the form of source code, object code, executable file or some intermediate form. The computer readable medium at least includes any entity or device capable of carrying computer program code, recording medium, computer memory, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), electrical carrier signal, telecommunication signal and software distribution medium. For example, U disk, mobile hard disk, magnetic disk or optical disk, etc. In some jurisdictions, according to legislation and patent practice, computer readable medium cannot be electrical carrier signal and telecommunication signal.

[0147] The present application realizes all or part of the processes in the above-mentioned embodiment methods, which can also be completed by a computer program product. When the computer program product runs on the computer equipment, it makes the computer equipment execute the steps in the above-mentioned embodiment methods.

[0148] In the above-mentioned embodiments, the description of each embodiment has its own emphasis, and the parts not described or recorded in detail in a certain embodiment can be referred to the related description of other embodiments.

[0149] Those skilled in the art can understand that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0150] In the embodiments provided by the present application, it should be understood that the disclosed apparatus / computer device and method can be implemented in other manners. For example, the described apparatus / computer device embodiments are merely schematic. For example, the division of the modules or units can be different, and each can include a plurality of sub-units. Some or all of the modules or units can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections can be indirect couplings or communication connections through some interfaces, devices or units, and can be in electrical, mechanical or other forms.

[0151] The units described as separated components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e. can be located in one place, or can be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purposes of the embodiments of the present application.

[0152] The above embodiments are only used to illustrate the technical solutions of the present application, rather than limit them. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced by equivalent ones. Such modifications or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A method for video revisit based on artificial intelligence, characterized in that, The video follow-up method comprises: obtaining a reference image of a target user, extracting image features of the reference image to obtain reference features; initializing an iteration number as zero, mapping the iteration number to a collection time point, collecting a real-time image of the target user at the collection time point, extracting real-time features of the real-time image, and calculating the similarity between the reference features and the real-time features; after calculating the similarity between the reference features and the real-time features, further comprising: when the similarity is less than or equal to a preset similarity threshold, increasing a first abnormality statistical value by one, and the initial value of the first abnormality statistical value is zero; when the similarity is greater than the preset similarity threshold, inputting the real-time features into a trained detection model for face detection, and outputting at least one face detection result; counting the number of all face detection results, when the number is the same as a preset number reference value, sending a follow-up question corresponding to the iteration number to the target user, and receiving a follow-up answer corresponding to the follow-up question; after counting the number of all face detection results, further comprising: when the number is not the same as the preset number reference value, increasing a second abnormality statistical value by one, and the initial value of the second abnormality statistical value is zero; increasing the iteration number by one, returning to execute the step of mapping the iteration number to the collection time point, until all follow-up questions have been sent to the target user, or the ratio of the first abnormality statistical value to the iteration number is less than a preset reference ratio, or the second abnormality statistical value is the same as a preset abnormality reference value, stopping iteration, and obtaining at least one set of follow-up questions and follow-up answers with a corresponding relationship; the step of mapping the iteration number to the collection time point comprises: obtaining a question and answer termination time point, which represents the time when the follow-up answer is received in the previous iteration process; multiplying the iteration number by a preset coefficient, calculating the sum of the multiplication result and the question and answer termination time point, and determining the calculation result as the collection time point.

2. The video revisit method of claim 1, wherein, The step of obtaining the reference image of the target user comprises: searching for a registered image corresponding to the target user from a preset database, the database comprising N sets of users and registered images with a corresponding relationship, N being an integer greater than zero; obtaining a snapshot image of the target user at a preset time point, performing similarity calculation on the snapshot image and the registered image, and determining the snapshot image as the reference image when the similarity calculation result is greater than a preset reference threshold.

3. The video visitation method of any of claims 1 to 2, wherein, The step of receiving the follow-up answer corresponding to the follow-up question comprises: determining a question time point at which the follow-up question is sent to the target user, starting from the question time point, receiving speech data of the target user frame by frame until the difference between the current frame time point and the question time point is greater than a preset difference threshold, stopping receiving, and obtaining all speech data; splicing all speech data, inputting the splicing result into a trained speech recognition model, and outputting the follow-up answer corresponding to the follow-up question.

4. The video revisit method of claim 3, wherein, The speech data of the target user is received frame by frame from the questioning time point until a difference between the current frame time point and the questioning time point is greater than a preset difference threshold, and the receiving is stopped, to obtain all speech data, including: Speech data of the target user is received frame by frame from the questioning time point, and when K continuous frames of speech data are noise data, it is determined that a preset condition is met; When the preset condition is met or the difference between the current frame time point and the questioning time point is greater than a preset difference threshold, the receiving is stopped, to obtain all speech data. 5.A video revisit device based on artificial intelligence, characterized in that, The video revisit device comprises: A feature extraction module is configured to obtain a reference image of a target user, extract image features of the reference image, and obtain reference features; A similarity calculation module is configured to initialize an iteration number as zero, map the iteration number to a collection time point, collect a real-time image of the target user at the collection time point, extract real-time features of the real-time image, and calculate a similarity between the reference features and the real-time features; The video revisit device further comprises: A first anomaly statistics module is configured to increase a first anomaly statistics value by one when the similarity is less than or equal to a preset similarity threshold, and the initial value of the first anomaly statistics value is zero; A face detection module is configured to input the real-time features into a trained detection model for face detection when the similarity is greater than a preset similarity threshold, and output at least one face detection result; A question sending module is configured to count a number of all face detection results, send a revisit question corresponding to a question serial number matched with the iteration number to the target user when the number is the same as a preset number reference value, and receive a revisit answer corresponding to the revisit question; The question sending module further comprises: A second anomaly statistics unit is configured to increase a second anomaly statistics value by one when the number is not the same as the preset number reference value, and the initial value of the second anomaly statistics value is zero; An iteration termination module is configured to increase the iteration number by one, return to execute the step of mapping the iteration number to the collection time point, until all revisit questions have been sent to the target user, or a ratio of the first anomaly statistics value to the iteration number is less than a preset reference ratio, or the second anomaly statistics value is the same as a preset anomaly reference value, and the iteration is stopped, to obtain at least one set of revisit questions and revisit answers having a corresponding relationship; The similarity calculation module comprises: A time point acquisition unit is configured to acquire a question and answer termination time point, and the question and answer termination time point is used to represent a time when the revisit answer is received in a previous iteration process; A time point determination unit is configured to multiply the iteration number by a preset coefficient, calculate a sum of the multiplication result and the question and answer termination time point, and determine the calculation result as the collection time point.

6. A computer device, comprising: The computer device comprises a processor, a memory, and a computer program stored in the memory and executable on the processor, and the processor executes the computer program to implement the video revisit method according to any one of claims 1 to 4.

7. A computer-readable storage medium storing a computer program, wherein the computer program comprises the following steps of: receiving a request for a resource from a client; determining whether the client is authorized to access the resource; and if the client is authorized to access the resource, providing the resource to the client. The computer program, when executed by a processor, implements the video revisit method according to any one of claims 1 to 4.

Images