Face processing method, device and system for convergence media video

Abstract

The invention relates to a face processing method, device and system for a convergence media video. Comprising the following steps: determining a current to-be-tested video from a plurality of obtained to-be-tested videos, and calculating a current hash value of the current to-be-tested video; comparing the current hash value with hash values corresponding to historical videos to be detected in the videos to be detected, if it is detected that the current hash value is different from the hash values of all the historical videos to be detected, extracting a key frame to be detected from the current video to be detected, and identifying a face area of an object to be detected in the key frame to be detected; extracting a face feature vector of the face region, and matching the face feature vector with a target object feature vector in a preset target object database; and when it is detected that matching succeeds, recording time information of the to-be-detected object in the to-be-detected video, and storing a matching result of the to-be-detected object and the time information. By adopting the method, the character object in the video can be checked efficiently and accurately.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a method, device and system for processing faces in convergent media videos. Background Art

[0002] With the rapid development of the media industry and the rise of short videos, a large number of videos of varying quality have emerged. In order to ensure the quality of videos, video review is particularly important. The traditional method of video review is mostly manual review, but the workload of manual review is large, and the review speed is very limited, which is difficult to match the update speed of the video. In particular, when reviewing the target person in the video, the accuracy of manual review is low.

[0003] Currently, no effective solution has been proposed for the problem that the efficiency and accuracy of reviewing human objects in videos in the prior art are low. Summary of the invention

[0004] Based on this, it is necessary to provide a face processing method, device and system for converged media videos in response to the above technical problems.

[0005] In the first aspect, the present application provides a method for face processing in converged media videos, including: Determine a current video to be tested from the acquired multiple videos to be tested, and calculate a current hash value of the current video to be tested; Compare the current hash value with the hash value corresponding to the historical videos to be tested in the video to be tested. If it is detected that the current hash value is different from the hash values ​​of all the historical videos to be tested, extract the key frame to be tested from the current video to be tested, and identify the face area of ​​the object to be tested in the key frame to be tested; extract the face feature vector of the face area, and match the face feature vector with the target object feature vector in the preset target object database; When a successful match is detected, the time information of the object to be tested included in the video to be tested is recorded, and the matching result and time information of the object to be tested are saved.

[0006] In one embodiment, obtaining a target object database includes: Acquire metadata corresponding to a preset target object and a facial feature vector of the target object, wherein the metadata includes identification information of the target object; When a processing instruction for a target object is obtained, metadata of the target object is queried based on the processing instruction, and a corresponding facial feature vector of the target object is obtained based on the query result of the metadata, wherein the processing instruction includes a modification instruction, a deletion instruction, and a query instruction.

[0007] In one embodiment, when a match is detected to be successful, the time information of the object to be tested is recorded in the video to be tested, and the matching result and time information of the object to be tested are saved, and the method further includes: Based on the matching result and time information, video editing processing is performed on the corresponding face area, wherein the video editing processing includes at least one of the following processing: deletion processing, replacement processing, and generating a warning prompt.

[0008] In one embodiment, after determining a current video to be tested from the acquired multiple videos to be tested and calculating a current hash value of the current video to be tested, the method further includes: The current hash value is compared with the hash value corresponding to the historical target video to be tested in the video to be tested. If it is detected that the current hash value is the same as the hash value of the historical target video to be tested in the historical video to be tested, the processing of the current video to be tested is terminated.

[0009] In one embodiment, identifying a face region of a subject to be tested in a key frame to be tested and extracting a face feature vector of the face region includes: Performing Fourier transform on the sub-image of the face region to obtain a clarity result corresponding to the sub-image of the face region; The clarity result is compared with a preset clarity threshold, and the facial feature vector is extracted from the sub-image of the face area whose clarity result is greater than the clarity threshold.

[0010] In one embodiment, identifying a face region of a subject to be tested in a key frame to be tested and extracting a face feature vector of the face region includes: If it is detected that the size of the sub-image of the face region is greater than a preset size threshold, a face feature vector is extracted from the sub-image of the face region.

[0011] In one embodiment, matching the facial feature vector with a target object feature vector in a preset target object database includes: Performing average processing on the target object feature vectors to obtain a target average feature vector corresponding to the target object in the target object database; The cosine distance between the target average feature vector and the face feature vector is calculated, and when it is detected that the cosine distance is less than a preset distance threshold, the match is determined to be successful.

[0012] In a second aspect, the present application also provides a face processing device for integrated media video. The device includes: An acquisition module, used to determine a current video to be tested from the acquired multiple videos to be tested, and calculate a current hash value of the current video to be tested; A calculation module is used to compare the current hash value with the hash values ​​corresponding to the historical videos to be tested in the video to be tested. If it is detected that the current hash value is different from the hash values ​​of all the historical videos to be tested, a key frame to be tested is extracted from the current video to be tested, and the face area of ​​the object to be tested in the key frame to be tested is identified; a face feature vector of the face area is extracted, and the face feature vector is matched with a target object feature vector in a preset target object database; The generation module is used to record the time information of the object to be tested in the video to be tested when a successful match is detected, and save the matching result and time information of the object to be tested.

[0013] In a third aspect, the present application further provides a face processing system for converged media videos, comprising: a video acquisition module and a face processing device for converged media videos as described above; A video acquisition module, used to determine a current video to be tested from multiple videos to be tested, and calculate a current hash value of the current video to be tested; The face processing device is used to execute the steps of the face processing method for the integrated media video as described above based on the current video to be tested, multiple videos to be tested and the current hash value.

[0014] In a fourth aspect, the present application further provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program: Determine a current video to be tested from the acquired multiple videos to be tested, and calculate a current hash value of the current video to be tested; Compare the current hash value with the hash value corresponding to the historical videos to be tested in the video to be tested. If it is detected that the current hash value is different from the hash values ​​of all the historical videos to be tested, extract the key frame to be tested from the current video to be tested, and identify the face area of ​​the object to be tested in the key frame to be tested; extract the face feature vector of the face area, and match the face feature vector with the target object feature vector in the preset target object database; When a successful match is detected, the time information of the object to be tested included in the video to be tested is recorded, and the matching result and time information of the object to be tested are saved.

[0015] The above-mentioned method, device and system for face processing of integrated media video first determine the current video to be tested from the multiple videos to be tested that have been acquired, and calculate the current hash value of the current video to be tested. The current hash value is compared with the hash value corresponding to the historical videos to be tested in the video to be tested. If it is detected that the current hash value is different from the hash values ​​of all the historical videos to be tested, the key frame to be tested is extracted from the current video to be tested, and the face area of ​​the object to be tested in the key frame to be tested is identified, the face feature vector of the face area is extracted, and the face feature vector is matched with the target object feature vector in the preset target object database. When a successful match is detected, the time information of the object to be tested included in the video to be tested is recorded, and the matching result and time information of the object to be tested are saved. Through this application, before processing the current video to be tested, the hash value of the historical video to be tested can be comprehensively used to detect whether the current video to be tested needs to be processed, so as to avoid repeated processing of a video and waste of computing power. Furthermore, the present application also includes matching the facial feature vector of the facial area with the target object feature vector in a preset target object database, thereby quickly generating the matching result of the object to be tested in the video to be tested and information such as time. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A diagram showing an application environment of a face processing method in one embodiment; Figure 2 is a schematic flow chart of a face processing method in one embodiment; Figure 3 is a flowchart of a face processing method in a preferred embodiment; Figure 4 is a structural block diagram of a face processing device in one embodiment; Figure 5 is a structural block diagram of a face processing system in one embodiment; Figure 6 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0018] The face processing method for converged media video provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. First, determine the current video to be tested from the multiple videos to be tested that have been acquired, and calculate the current hash value of the current video to be tested. Compare the current hash value with the hash value corresponding to the historical video to be tested in the video to be tested. If it is detected that the current hash value is different from the hash value of all historical videos to be tested, extract the key frame to be tested from the current video to be tested, and identify the face area of ​​the object to be tested in the key frame to be tested, extract the face feature vector of the face area, and match the face feature vector with the target object feature vector in the preset target object database. When a successful match is detected, record the time information of the object to be tested in the video to be tested, and save the matching result and time information of the object to be tested. The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. The portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers.

[0019] In one embodiment, Figure 2 As shown, a face processing method for integrated media video is provided, and the method is applied to Figure 1 The server in the example is used to illustrate the following steps: Step S210, determining a current video to be tested from the acquired multiple videos to be tested, and calculating a current hash value of the current video to be tested.

[0020] Specifically, a plurality of videos to be tested are first obtained. In practical applications, videos are usually pulled from the FTP file system (File Transfer Protocol) of the current tenant. In some preferred embodiments, all videos to be tested need to be detected and face processed. Therefore, before processing the videos to be tested, a task queue can be established based on all videos to be tested, and detection and face processing can be performed in sequence according to the task queue. The above-mentioned current video to be tested is the video to be tested that currently needs to be detected and face processed. Calculate the current hash value of the current video to be tested, that is, perform MD5 (Message-Digest Algorithm 5) verification on the current video to be tested. In practical applications, the Python standard library hashlib can be used to calculate the MD5 hash value of the video file, that is, calculate the current hash value of the current video to be tested.

[0021] Step S220, compare the current hash value with the hash value corresponding to the historical videos to be tested in the video to be tested. If it is detected that the current hash value is different from the hash values ​​of all the historical videos to be tested, extract the key frame to be tested from the current video to be tested, and identify the face area of ​​the object to be tested in the key frame to be tested; extract the face feature vector of the face area, and match the face feature vector with the target object feature vector in a preset target object database.

[0022] Specifically, since each video in the video to be tested needs to be detected and face processed, before processing the current video to be tested, there are multiple historical videos to be tested whose faces have been processed. After calculating the current hash value of the current video to be tested, the current hash value is compared with the hash value corresponding to the historical video to be tested to detect whether the current hash value of the current video to be tested already exists in the database. If it is detected that the current hash value is different from the hash value of the historical video to be tested, it means that the current hash value does not exist in the database, indicating that the current video to be tested has never been processed, and the current video to be tested can be subjected to subsequent processing. Similarly, if it is detected that the current hash value is the same as the hash value of a historical video to be tested, it means that the current video to be tested has been processed before, and in this embodiment, no secondary face processing is required, and the processing process of the current video to be tested can be terminated at this time.

[0023] If it is detected that the current hash value is different from the hash values ​​of all historical videos to be tested, the key frame to be tested is extracted from the current video to be tested, and face detection is performed on the key frame to be tested. There is no need to perform face detection processing on each frame in the video, which effectively reduces the amount of calculation. In this embodiment, FFmpeg can be used to extract frames from the current video to be tested, or the key frame to be tested in the current video to be tested can be extracted at a specified time interval (such as extracting one frame every second), and the extracted key frame to be tested is saved to a specific directory on the server.

[0024] After obtaining the key frame to be tested in the current video to be tested, the face area of ​​the object to be tested in the key frame to be tested is identified. In some preferred examples, there are multiple methods for identifying the face area in the key frame, and both traditional target detection algorithms and face recognition based on deep learning neural networks are acceptable. After the face area is detected, the coordinates of the detected face position can be recorded for each key frame to be tested, such as the upper left corner and lower right corner coordinates of the detected face position, so as to obtain the sub-image where the face area of ​​the object to be tested is located. Further, the face feature vector of the face area is extracted. In practical applications, the pre-trained model in the dlib library (Dlib C++ Library) can be used to extract the 128-dimensional face feature vector of the face, and the extracted face feature vector is matched with the target object feature vector in the preset target object database, wherein a similarity threshold can be preset. When the similarity between the face feature vector in the video and the target object feature vector in the database is greater than or equal to the preset similarity threshold, the match is judged to be successful.

[0025] In some preferred embodiments, the present application is preferably applied to the scene of deleting or blocking the face of a person who needs to be blocked in the video. Accordingly, the target object database stores the target object feature vector of the face of the preset object (i.e., the preset person who needs to be blocked) and the metadata corresponding to the target object. Furthermore, the content contained in the target object database of the present application can be modified according to actual needs. For example, when it is necessary to block or block the vehicle in the video, the content stored in the target object database can be the metadata of different types of vehicles and the feature vector of the preset vehicle, etc.

[0026] Step S230, when a successful match is detected, the time information of the object to be tested included in the video to be tested is recorded, and the matching result and time information of the object to be tested are saved.

[0027] Specifically, when the match is successful, the matching result can be fed back to the main program through asynchronous callback, and the time information of the object to be tested in the current video to be tested, such as time period information, etc., is recorded, and a detailed report is generated accordingly, such as the matching result, the name of the person to be blocked, the matching degree, the timestamp, etc. In actual applications, the report can be exported to a specified format (such as CSV, PDF, etc.).

[0028] In some preferred embodiments, if an error occurs, such as network failure, service unavailability, etc., the present application may also provide a retry mechanism to ensure that the face processing tasks of all videos to be tested can be processed correctly.

[0029] Through step S210 to step S230, the present application can first determine whether the current video to be tested has been processed by the hash value. If the video has been processed, it can be discovered in time to avoid invalid secondary processing. In practical applications, the present application is usually used to process large batches or long-time videos. Therefore, the detection of whether the current video to be tested has been processed can effectively save a lot of computing resources. Furthermore, the present application matches the facial feature vector of the face area with the target object feature vector in the preset database, which can quickly determine the matching of the face in the video without manually identifying the face appearing in the video. It not only effectively improves the matching and processing speed, but also greatly improves the accuracy of face matching processing.

[0030] In some embodiments, obtaining a target object database includes: Acquire metadata corresponding to a preset target object and a facial feature vector of the target object, wherein the metadata includes identification information of the target object; When a processing instruction for a target object is obtained, metadata of the target object is queried based on the processing instruction, and a corresponding facial feature vector of the target object is obtained based on the query result of the metadata, wherein the processing instruction includes a modification instruction, a deletion instruction, and a query instruction.

[0031] Specifically, the present application needs to pre-set the target object database according to actual needs, first obtain the metadata corresponding to the target object and the target object facial feature vector, wherein the target object is a preset person who needs to be shielded, and accordingly, the metadata corresponding to the target object is preferably set to the basic information of the target object, such as the name, ID, picture, etc. of the person who needs to be shielded, and the metadata corresponding to the target object is saved to the MySQL database. Further, the target object facial feature vector is the image information corresponding to the target object (i.e., the preset person who needs to be shielded), extract the facial feature vector of the image information corresponding to the target object, obtain the target object facial feature vector, and store the target object facial feature vector in the Milvus database. In some preferred embodiments, before extracting the target object facial feature vector, the acquired image of the person who needs to be shielded can be pre-processed, such as scaling, cropping, etc., to improve the quality of the features extracted subsequently. It can be further understood that although the metadata and the target object facial feature vector are not saved together, the two are in a binding relationship, that is, the metadata and the target object facial feature vector of the same target object are in a binding relationship. Similarly, the above method is also applicable to the process of adding new target objects acquired after the target object database is established.

[0032] After the target object database is established, the target object database can be added, deleted, modified and queried. In order to improve the efficiency of addition, deletion, modification and query, when a processing instruction for a target object in the database is obtained, the metadata of the target object is first queried. After the metadata is queried, the facial feature vector of the target object is queried according to the metadata, thereby completing the modification, deletion, query and other operations.

[0033] Specifically, when modifying the information of the target object, first obtain the modification instruction, and query the information of the target object according to the ID or other identifier, and then update the metadata corresponding to the target object accordingly. If the image changes, query the facial feature vector of the target object according to the metadata, re-extract the image features, and update the facial feature vector of the target object.

[0034] Similarly, when deleting the target object's information, first obtain the deletion instruction, and query the target object's information according to the ID or other identifier, and then delete the metadata corresponding to the target object and the corresponding target object's facial feature vector accordingly.

[0035] Similarly, when querying the information of the target object, the query instruction is first obtained, and the target object information is queried according to the ID or other identifier, and then the target object facial feature vector corresponding to the target object is queried accordingly.

[0036] In some embodiments, the method further comprises: Based on the matching result and time information, video editing processing is performed on the corresponding face area, wherein the video editing processing includes at least one of the following processing: deletion processing, replacement processing, and generating a warning prompt.

[0037] Specifically, considering that the present application is preferably used in the shielding process of the persons who need to be shielded in the video, after the successful matching result of the persons who need to be shielded and the corresponding time information are saved, the face area of ​​the successfully matched persons who need to be shielded needs to be edited accordingly. The video editing process includes, but is not limited to, deleting the area where the persons who need to be shielded are located, replacing the area where the persons who need to be shielded are located (such as replacing the background content with the area where the persons who need to be shielded are located), generating an alarm prompt for the area where the persons who need to be shielded are located, etc. Furthermore, for the newly added target object (i.e., the person who needs to be shielded), the processed videos to be tested can be automatically scanned to find and process the segments where the newly added persons who need to be shielded are located in each historical video to be tested.

[0038] In some embodiments, the method further comprises: The current hash value is compared with the hash value corresponding to the historical target video to be tested in the video to be tested. If it is detected that the current hash value is the same as the hash value of the historical target video to be tested in the historical video to be tested, the processing of the current video to be tested is terminated.

[0039] Specifically, if it is detected that the current hash value of the current video to be tested is the same as the hash value corresponding to the historical video to be tested in the video to be tested, it means that the current video to be tested has been processed before, and no secondary processing is required in this application. Therefore, if it is detected that the current hash value is the same as the hash value of the historical video to be tested, the detection and processing of the current video to be tested is terminated and skipped.

[0040] In some embodiments, the method further comprises: Performing Fourier transform on the sub-image of the face region to obtain a clarity result corresponding to the sub-image of the face region; The clarity result is compared with a preset clarity threshold, and the facial feature vector is extracted from the sub-image of the face area whose clarity result is greater than the clarity threshold.

[0041] Specifically, the sub-image of the face region is subjected to Fourier transformation, and the spectral center intensity of the sub-image of the face region is calculated, which is used as an indicator of image clarity, that is, the above-mentioned clarity threshold is determined by the spectral center intensity. The sub-image of the face region is subjected to Fourier transformation, and a clarity result corresponding to the sub-image of the face region is obtained. The clarity result is compared with the clarity threshold, and the sub-image of the face region whose clarity result is greater than the clarity threshold is retained, and the face feature vector is extracted from the sub-image whose clarity result is greater than the clarity threshold. Through this embodiment, the clarity of the face region can be detected before extracting the face feature vector. It can be understood that if the clarity of the sub-image of the face region is too low, there is no need to perform subsequent operations such as deletion and replacement on the face region in actual applications, because when the user watches the video, due to the low clarity, it will not bring a bad user experience. Therefore, this embodiment only performs subsequent processing operations on the sub-image whose clarity is greater than the clarity threshold, which can effectively improve the efficiency of video processing and greatly reduce the amount of calculation.

[0042] In some embodiments, the method further comprises: If it is detected that the size of the sub-image of the face region is greater than a preset size threshold, a face feature vector is extracted from the sub-image of the face region.

[0043] Specifically, the size of the sub-image of the face area is detected. If the size of the sub-image of the face area is detected to be larger than a preset size threshold, the subsequent facial feature vector is extracted from the sub-image of the face area, wherein the size threshold can be set according to actual needs, such as 100×100 pixels. Similarly, if the size of the sub-image of the face area is detected to be less than or equal to the preset size threshold, the processing of the sub-image is terminated and skipped. Through this embodiment, the size screening of the sub-image of the face area can be completed. Considering that if the size of the sub-image of the face area of ​​the person to be shielded is too small, it will basically not affect the user's viewing experience. Therefore, in order to improve computing efficiency and reduce the consumption of computing resources, the processing of the sub-image of the face area whose size is less than or equal to the size threshold can be terminated and skipped.

[0044] In some embodiments, the method further comprises: Performing average processing on the target object feature vectors to obtain a target average feature vector corresponding to the target object in the target object database; The cosine distance between the target average feature vector and the face feature vector is calculated, and when it is detected that the cosine distance is less than a preset distance threshold, the match is determined to be successful.

[0045] Specifically, the target object feature vectors of each target object preset in the database are averaged to obtain the target average feature vector corresponding to the target object. In practical applications, it is preferred to use the Milvus client to query the feature vector data group of the target object (i.e., the person to be shielded) from the target object database, and average the acquired feature vector data group to obtain the target average feature vector of the target object.

[0046] Further, the cosine distance between the target average feature vector and the face feature vector in the target video to be tested is calculated, wherein the method for calculating the cosine distance includes but is not limited to first calculating the dot product of two vectors, namely the target average feature vector and the face feature vector. The dot product is the sum of the products of the corresponding elements of the vectors, which measures the extent to which the two vectors "point" in the same direction. Then, two loud Euclidean moduli (or lengths) are calculated respectively. The modulus is the square root of the sum of the squares of all elements in the vector, which measures the size of the vector. The cosine similarity between the two vectors is calculated by dividing the dot product by the product of the moduli of the two vectors. In practical applications, the value of the cosine similarity is generally between -1 and 1, wherein 1 indicates that the two vectors are completely in the same direction, 0 indicates that the two vectors are orthogonal (i.e., vertical), and -1 indicates that the two vectors are completely opposite. Finally, the cosine similarity is subtracted from 1 to obtain the above cosine distance.

[0047] If the detected cosine distance is less than the preset distance threshold, the match is successful, and it is determined that the face in the target video is the target object in the database.

[0048] Figure 3 The figure is a flowchart of a face processing method for converged media video in a preferred embodiment.

[0049] Step S310, performing MD5 verification on the current video to be tested, wherein the MD5 verification includes, first, the user uploads the current video file to be tested through the API, and saves the current video file to be tested to a temporary directory on the server, and performs MD5 verification on the current video to be tested, that is, using the Python standard library hashlib to calculate the current hash value of the video file, and detecting whether the current hash value is the same as the hash value corresponding to the historical video to be tested. If they are the same, then end the processing of the current video to be tested, if they are different, jump to step S320.

[0050] Step S320 , extracting key frames to be tested from the current video to be tested at a specified time interval (such as extracting one frame every second), and saving the extracted key frames to be tested to a specific directory on the server.

[0051] Step S330, performing face detection on each key frame to be tested extracted, wherein the YUNET model can be used to perform face detection on each key frame to be tested, and the face position coordinates of the detected face area are recorded, the face area can be recorded by the upper left corner and lower right corner coordinates, and the sub-image where the face area is located is extracted.

[0052] Step S340, determining whether the clarity of the sub-image in the face area is higher than a preset clarity threshold, wherein the sub-image in the face area is first subjected to Fourier transform, and then the spectral center intensity of the sub-image is calculated, which is used as the above-mentioned clarity threshold. If it is detected that the clarity of the sub-image is higher than the clarity threshold, the process jumps to step S350; if it is detected that the clarity of the sub-image is lower than or equal to the clarity threshold, the process ends and skips the processing of the sub-image.

[0053] Step S350, determine whether the size of the sub-image of the face area is greater than a preset size threshold. If it is detected that the size of the sub-image is greater than the size threshold, jump to step S360. If it is detected that the size of the sub-image is less than or equal to the size threshold, end and skip processing of the sub-image.

[0054] Step S360, face detection is performed again on the sub-image of the filtered face area, wherein the Face_recognition model can be used for face detection. If it is detected that the sub-image does contain a face, jump to step S370. If it is detected that the sub-image does not contain a face, the processing of the sub-image is terminated.

[0055] Step S370, extracting the facial feature vector of the sub-image of the facial region.

[0056] Step S380, matching the extracted facial feature vector with each target object feature vector in the database, and completing the detection of the current video to be tested after a successful match is detected.

[0057] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0058] Based on the same inventive concept, the embodiment of the present application also provides a face processing device for converged media videos for implementing the face processing method for converged media videos involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more embodiments of the face processing device for converged media videos provided below can refer to the limitations of the face processing method for converged media videos above, and will not be repeated here.

[0059] In one embodiment, Figure 4 As shown, a face processing device for converged media video is provided, including: an acquisition module 41, a calculation module 42 and a generation module 43, wherein: An acquisition module 41 is used to determine a current video to be tested from the acquired multiple videos to be tested, and calculate a current hash value of the current video to be tested; The calculation module 42 is used to compare the current hash value with the hash value corresponding to the historical videos to be tested in the video to be tested. If it is detected that the current hash value is different from the hash values ​​of all the historical videos to be tested, a key frame to be tested is extracted from the current video to be tested, and the face area of ​​the object to be tested in the key frame to be tested is identified; a face feature vector of the face area is extracted, and the face feature vector is matched with a target object feature vector in a preset target object database; The generating module 43 is used to record the time information of the object to be tested included in the video to be tested when a successful match is detected, and save the matching result and time information of the object to be tested.

[0060] Each module in the above-mentioned face processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0061] In one embodiment, a face processing system for converged media video is also provided, including a video acquisition module and a face processing device for converged media video; A video acquisition module is used to determine a current video to be tested from multiple videos to be tested, and calculate a current hash value of the current video to be tested; The face processing device is used to perform the face processing steps described above based on the current video to be tested, multiple videos to be tested and the current hash value.

[0062] Specifically, Figure 5 FIG. 1 is a structural block diagram of a face processing system in an embodiment. First, a plurality of videos to be tested are obtained through a video acquisition module, and a current video to be tested that needs to be tested is determined from the plurality of videos to be tested. This embodiment also includes a plurality of face detection servers, such as Figure 5 The face detection server 1 and the face detection server 2 in the system, through the task scheduling processor, assign the current video to be tested to the corresponding server for subsequent face processing. The server that obtains the current video to be tested first detects the hash value of the current video to be tested. When it is detected that the current hash value is different from the hash value of the historical video to be tested, the key frame to be tested is extracted from the current video to be tested, and the face area of ​​the object to be tested in the key frame to be tested is identified. The server then matches the face feature vector of the face area with the feature vector of the target object in the target object database. When a successful match is detected, the time information of the object to be tested included in the current video to be tested is recorded, and the matching result and time information of the object to be tested are saved. The target object database includes metadata corresponding to the preset target object and the target object facial feature vector, and the metadata and the target object facial feature vector are in a binding correspondence relationship.

[0063] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 6As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data related to face processing. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a face processing method is implemented.

[0064] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0065] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0066] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0067] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0068] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A face processing method for converged media video, characterized in that: The method comprises: Determine a current video to be tested from the acquired multiple videos to be tested, and calculate a current hash value of the current video to be tested; The current hash value is compared with the hash values ​​corresponding to the historical videos to be tested in the video to be tested. If it is detected that the current hash value is different from the hash values ​​of all the historical videos to be tested, a key frame to be tested is extracted from the current video to be tested, and a face area of ​​the object to be tested in the key frame to be tested is identified; a face feature vector of the face area is extracted, and the face feature vector is matched with a target object feature vector in a preset target object database; When a successful match is detected, the time information of the object to be tested included in the video to be tested is recorded, and the matching result of the object to be tested and the time information are saved.

2. The method according to claim 1, characterized in that Acquiring the target object database includes: Acquire metadata corresponding to a preset target object and a facial feature vector of the target object, wherein the metadata includes identification information of the target object; When a processing instruction for the target object is obtained, the metadata of the target object is queried based on the processing instruction, and the corresponding facial feature vector of the target object is obtained based on the query result of the metadata, wherein the processing instruction includes a modification instruction, a deletion instruction, and a query instruction.

3. The method according to claim 1, characterized in that When a successful match is detected, the time information of the object to be tested is recorded in the video to be tested, and the matching result of the object to be tested and the time information are saved, and the method further includes: Based on the matching result and the time information, a video editing process is performed on the corresponding face area, wherein the video editing process includes at least one of the following processes: deletion process, replacement process, and generation of a warning prompt.

4. The method according to claim 1, characterized in that After determining the current video to be tested from the acquired multiple videos to be tested and calculating the current hash value of the current video to be tested, the method further includes: The current hash value is compared with the hash value corresponding to the historical target video to be tested in the video to be tested. If it is detected that the current hash value is the same as the hash value of the historical target video to be tested in the historical video to be tested, the processing of the current video to be tested is terminated.

5. The method according to claim 1, characterized in that The step of identifying a face region of the subject to be tested in the key frame to be tested and extracting a face feature vector of the face region includes: Performing Fourier transform on the sub-image of the face region to obtain a clarity result corresponding to the sub-image of the face region; The clarity result is compared with a preset clarity threshold, and a facial feature vector is extracted from a sub-image of the face region whose clarity result is greater than the clarity threshold.

6. The method according to claim 1, characterized in that The step of identifying a face region of the subject to be tested in the key frame to be tested and extracting a face feature vector of the face region includes: If it is detected that the size of the sub-image of the face region is greater than a preset size threshold, a face feature vector is extracted from the sub-image of the face region.

7. The method according to claim 1, characterized in that The matching of the facial feature vector with a target object feature vector in a preset target object database includes: Performing an average processing on the target object feature vector to obtain a target average feature vector corresponding to the target object in the target object database; The cosine distance between the target average feature vector and the face feature vector is calculated, and when it is detected that the cosine distance is less than a preset distance threshold, it is determined that the match is successful.

8. A face processing device for integrated media video, characterized in that: The device comprises: An acquisition module, used to determine a current video to be tested from the acquired multiple videos to be tested, and calculate a current hash value of the current video to be tested; A calculation module, used for comparing the current hash value with the hash values ​​corresponding to the historical videos to be tested in the video to be tested, and if it is detected that the current hash value is different from the hash values ​​of all the historical videos to be tested, extracting the key frame to be tested from the current video to be tested, and identifying the face area of ​​the object to be tested in the key frame to be tested; extracting the face feature vector of the face area, and matching the face feature vector with the target object feature vector in a preset target object database; The generating module is used to record the time information of the object to be tested included in the video to be tested when a successful match is detected, and save the matching result of the object to be tested and the time information.

9. A face processing system for converged media videos, characterized in that: The system comprises: a video acquisition module and the face processing device for integrated media video according to claim 8; The video acquisition module is used to determine a current video to be tested from multiple videos to be tested, and calculate a current hash value of the current video to be tested; The face processing device is used to perform the steps according to any one of claims 1 to 7 based on the current video to be tested, multiple videos to be tested and the current hash value.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

Images