Target object detection method and device, terminal equipment and computer readable storage medium

By processing the facial images of candidate subjects and constructing feature sequences, an object detection model is trained, which solves the problems of accuracy and stability in speaker detection in video scenes, and achieves more accurate speaker recognition and stable video tracking.

CN118942128APending Publication Date: 2024-11-12TCL TECHNOLOGY GROUP CORPORATION
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202310542415.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-12
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

In video scenarios such as remote meetings and online training, existing technologies struggle to accurately detect speakers using face detection models, leading to frequent video feed changes and target detection errors.

Method used

By processing the facial images of candidate objects, extracting mouth feature information, constructing initial and fused feature sequences, and combining the feature sequences of historical and current time windows, an object detection model is trained to predict the probability and state of the speaking target object.

Benefits of technology

It improves the accuracy and stability of detecting speaking targets in video scenes, and reduces video frame switching and detection errors.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118942128A_ABST
    Figure CN118942128A_ABST
Patent Text Reader

Abstract

The invention provides a target object detection method and device, terminal equipment and a computer readable storage medium, and the method comprises the steps: carrying out the processing of a face image of each candidate object, and obtaining the feature information of each candidate object; and processing the feature information, and detecting a speaking target object from each candidate object. According to the invention, the accuracy and stability of detecting the speaking target object in the video scene can be effectively improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a target object detection method, apparatus, terminal device and computer-readable storage medium. Background Art

[0002] In video scenarios such as remote meetings and online training, professional cameramen are usually required to judge the speakers in the video and to track and detect the speakers in real time. However, with the development of machine learning, face detection models are now mostly used to detect the target speakers by detecting the mouth state of the characters in the video, saving the cost and time of hiring professional cameramen.

[0003] However, due to the defects such as the lip movement information of the mouth state is difficult to capture, the movements are irregular, and there is no strong correlation with the speaking movements, in real video scenes, most face detection models not only find it difficult to accurately achieve target detection, but also have defects such as frequent video screen switching and target detection errors. Summary of the invention

[0004] The present application provides a target object detection method, which can effectively improve the accuracy and stability of detecting a speaking target object in a video scene.

[0005] In a first aspect, the present application provides a target object detection method, the method comprising:

[0006] Processing the facial image of each candidate object to obtain feature information of each candidate object;

[0007] The feature information is processed to detect the speaking target object from each candidate object.

[0008] In a second aspect, the present application also provides a face detection model training device, the device comprising:

[0009] A face image processing module is used to process the face image of each candidate object to obtain feature information of each candidate object;

[0010] The target object detection module is used to process the feature information and detect the target object of the speech from each candidate object.

[0011] In a third aspect, the present application also provides a terminal device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps in any one of the target object detection methods.

[0012] In a fourth aspect, the present application further provides a computer-readable storage medium, on which a computer program is stored, and the computer program is executed by a processor to implement the steps in any one of the target object detection methods.

[0013] The target object detection method, apparatus, terminal device and computer-readable storage medium provided in the present application process the facial image of the candidate object, extract relatively rich feature information of the candidate object, and detect the speaking target object based on the rich feature information, thereby effectively improving the accuracy and stability of detecting the speaking target object in the video scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0015] Figure 1 is a schematic diagram of a scene of a target object detection system provided in an embodiment of the present application;

[0016] Figure 2 This is a schematic diagram of a process flow of a target object detection method in an embodiment of the present application;

[0017] Figure 3 is a schematic diagram of a training target object detection model provided in an embodiment of the present application;

[0018] Figure 4 is a schematic diagram of the overall process of detecting a target object provided in an embodiment of the present application;

[0019] Figure 5 is a schematic diagram of mouth feature points provided in an embodiment of the present application;

[0020] Figure 6 It is a schematic diagram of a functional module of a face detection model training device in an embodiment of the present application;

[0021] Figure 7 It is a schematic diagram of the structure of the terminal device in the embodiment of the present application. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0023] In the description of the present application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the feature. In the description of the present application, "plurality" means two or more, unless otherwise clearly and specifically defined.

[0024] In this application, the word "exemplary" is used to mean "used as an example, illustration or description". Any embodiment described as "exemplary" in this application is not necessarily to be construed as being more preferred or advantageous than other embodiments. At the same time, it is to be understood that in the specific implementation of this application, when user information, user data and other related data are involved, when the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data shall comply with relevant laws, regulations and standards of relevant countries and regions.

[0025] In order to enable any person skilled in the art to implement and use the present application, the following description is provided. In the following description, details are listed for the purpose of explanation. It should be understood that those of ordinary skill in the art will recognize that the present application can be implemented without using these specific details. In other examples, known structures and processes will not be elaborated in detail to avoid unnecessary details that make the description of the present application obscure. Therefore, the present application is not intended to be limited to the embodiments shown, but is consistent with the widest range of principles and features disclosed in the present application.

[0026] The present application provides a target object detection method, apparatus, device and storage medium, which are described in detail below.

[0027] See also Figure 1 , Figure 1 Schematic diagram of a target object detection system provided in an embodiment of the present application. The target object detection system may include a terminal device 100 and a storage device 200. The storage device 200 may transmit data to the terminal device 100. Figure 1 The terminal device 100 in the embodiment can obtain the image stored in the storage device 200 to execute the target object detection method in the present application.

[0028] In the embodiment of the present application, the terminal device 100 includes but is not limited to a desktop computer, a portable computer, a network server, a PDA (Personal Digital Assistant), a tablet computer, a wireless terminal device, an embedded device, etc.

[0029] In an embodiment of the present application, communication between the terminal device 100 and the storage device 200 can be achieved through any communication method, including but not limited to mobile communications based on the 3rd Generation Partnership Project (3GPP), Long Term Evolution (LTE), and Worldwide Interoperability for Microwave Access (WiMAX), or computer network communications based on the TCP / IP Protocol Suite (TCP / IP) and User Datagram Protocol (UDP).

[0030] It should be noted that Figure 1 The scene diagram of the target object detection system shown is only an example. The target object detection system and scene described in the embodiment of the present application are intended to more clearly illustrate the technical solution of the embodiment of the present application, and do not constitute a limitation on the technical solution provided in the embodiment of the present application. Ordinary technicians in this field can know that with the evolution of the target object detection system and the emergence of new business scenarios, the technical solution provided in the embodiment of the present application is also applicable to similar technical problems.

[0031] like Figure 2 As shown, Figure 2 This is a flow chart of an embodiment of a target object detection method in an embodiment of the present application. The target object detection method may include the following steps 201 to 202:

[0032] Step 201: Process the facial image of each candidate object to obtain feature information of each candidate object.

[0033] Among them, the candidate object refers to the person object that appears in the video screen, and the method of the present application can achieve the effect of predicting the target object of the speech from multiple candidate objects. It can be understood that in video scenes such as remote meetings, online training, and online speeches, in order to ensure normal information exchange and communication, there will be at least one speaker in the above video scenes, and the speaker’s face will usually be detected, tracked, or close-up video effects will be performed to make the speaker as much as possible in the center of the video screen to highlight the speaker’s speaking effect and the offline user’s viewing experience.

[0034] The feature information of the candidate object may be a sequence of mouth features of each candidate object in the current time window. The mouth feature sequence refers to a linear feature sequence that characterizes the mouth motion state of the candidate object. In some embodiments, the mouth feature may be a mouth displacement, which may be calculated by the position change or distance between the mouth feature points. In some embodiments, a video processing library in a video processing software or programming language may be used to read a video file containing the candidate object into a frame image sequence of a face image, and then an object segmentation algorithm for processing the image, such as color segmentation, shape segmentation, or deep learning algorithms, is used to process each frame of the face image, separate the lip part of the candidate object from the face image, and obtain a mouth feature sequence corresponding to multiple frames of face images.

[0035] like Figure 5 As shown, for the mouth of each candidate object, multiple mouth feature points may be pre-set, and the position change of at least one mouth feature point in the video may be calculated to calculate the corresponding mouth displacement, for example, the mouth displacement is 1 mm, 2 mm, etc. In addition, two mouth feature points on the same horizontal line or vertical line may be pre-set as a pair of measurement points, and the mouth displacement is subsequently calculated by capturing the distance between each pair of measurement points. This embodiment does not limit the specific calculation method of the mouth displacement.

[0036] Among them, the time window refers to a technology that takes a continuous subsequence of a fixed length for processing in a time series data. In this application, assuming that the length of a time window is 1 second, 1 second can contain 30 frames of sequence data, then the current time window can be the current 1 second, and accordingly, the previous time window of the current time window can be the second before the current moment, and the next time window of the current time window can be the second after the current moment. For the length of a single time window, this embodiment does not limit this.

[0037] In some embodiments, the mouth feature may also be a mouth optical flow feature obtained by an optical flow method, for example, the mouth optical flow feature may be a feature of the instantaneous velocity, acceleration, angle, etc. of a certain feature point of the mouth. This embodiment does not limit the specific type of the mouth feature.

[0038] Step 202: Process the feature information and detect the speaking target object from each candidate object.

[0039] It can be understood that the method of the present application can detect the target object speaking from a video containing multiple candidate objects based on the feature information of multiple candidate objects. However, predicting the speaker in a video is a relatively complex task, which requires considering the influence of multiple factors, such as the speaking speed, mouth shape, speaking habits and emotional state of different people. Therefore, in practical applications, it is necessary to use more complex and sophisticated methods to process the feature information of candidate objects in order to improve the accuracy and stability of prediction.

[0040] Optionally, step 202 may further include:

[0041] Acquire a fused feature sequence of a historical time window, wherein the fused feature sequence represents a feature sequence of a speech state of each candidate object;

[0042] splicing the mouth feature sequence and the fusion feature sequence to obtain a spliced ​​feature sequence;

[0043] Based on the concatenated feature sequence, the target object of the speech is detected from each candidate object.

[0044] The fused feature sequence represents the feature sequence of the speaking state of each candidate object. The speaking state can be divided into two states: speaking and not speaking. The speaking state can be further divided into speaking and about to finish speaking. The not speaking state can be further divided into not speaking and about to speak. Therefore, compared with the mouth feature sequence, the fused feature sequence has richer feature information, which can improve the accuracy of the subsequent model in predicting the speaking target object and its speaking state.

[0045] It can be understood that in the above video scene, the speaker achieves speaking action through mouth movement, but it should be noted that although other candidate objects other than the speaker do not speak, they will also produce mouth movements due to yawning, sneezing, licking lips, opening mouth naturally, etc., so it is not possible to directly judge whether the candidate object is speaking by whether there is mouth displacement and the size of mouth displacement. Moreover, in the video screen, the mouth movement and mouth displacement are small in themselves, and the capture will be affected by the jitter and unclear factors of the video screen, resulting in errors in the value of the obtained mouth displacement. Therefore, based on the above factors, it is necessary to preprocess the mouth features to eliminate noise and redundant information in the features, and construct a mouth feature sequence and fusion feature sequence that is conducive to predicting the target object.

[0046] In order to better implement the target object detection method in the embodiment of the present application, before step 202, the method of the present application further includes:

[0047] Extract features from the facial image sequence to obtain an initial mouth feature sequence;

[0048] Splitting the initial mouth feature sequence into a first mouth feature sequence and a second mouth feature sequence;

[0049] Based on the speech state label of the first mouth feature sequence, the first mouth feature sequence is fused to obtain an initial fused feature sequence;

[0050] Splicing the initial fusion feature sequence with the second mouth feature sequence to obtain an initial spliced ​​feature sequence;

[0051] Based on the initial concatenated feature sequence, the prediction probability of speech is obtained;

[0052] The object detection model is trained according to the speech prediction probability and the speech probability label of the face image sequence to obtain a trained object detection model.

[0053] In some embodiments, a face image can be obtained from multiple consecutive video frames in a sample video, and an initial mouth feature sequence can be extracted from the face image. It can be understood that since any candidate object will perform continuous mouth opening and closing movements when speaking, the resulting mouth displacement is continuous and not fixed, and the context information of the above mouth features needs to be constructed in the order of the time window to improve the accuracy and stability of subsequent detection of the target object.

[0054] It should be noted that in order to ensure the training effect of the object detection model, the sample videos need to be manually labeled. For example, the continuous speaking time of the speaker in the sample video needs to reach a certain number of frames. After the frame positions of the start and end of the speaker's speech are determined by manual labeling, all the frames of continuous speech are normally transformed and used as labels for model training.

[0055] In some embodiments, multiple mouth features may be encoded by an object detection model to obtain an initial mouth feature sequence. Figure 3 , Figure 3 Schematic diagram of a training target object detection model provided in an embodiment of the present application. Figure 3 As shown, when training an object detection model, a plurality of initial mouth features may be first input into a linear processing unit of the object detection model for encoding.

[0056] First, multiple initial mouth features The corresponding time domain feature map will be extracted through the Res Block residual module layer To obtain the temporal order information between the initial mouth features from multiple different video frames. represents the time domain feature. The number of channels of the time domain feature map is C=256, F is the dimension corresponding to the initial mouth feature, and L is the number of frames corresponding to the initial mouth feature. Then, the time domain feature map M is globally pooled through the Global Pooling layer, and the mean and variance of the feature are extracted from each channel. Finally, the extracted mean and variance are processed into the corresponding linear feature sequence through the Linear layer to obtain the initial feature sequence.

[0057] In some embodiments, in order to construct the mouth movement states for starting and ending speech, the initial mouth feature sequence can be first split into a first mouth feature sequence and a second mouth feature sequence, and then based on the speech state label of the first mouth feature sequence, the first mouth feature sequence can be fused with the speech action to obtain an initial fused feature sequence.

[0058] Specifically, the speaking status label can be constructed The label values ​​are set to 1 and 0, where 1 represents the candidate speaking and 0 represents the speaker not speaking, so as to distinguish the two different speaking states. Where T = L / 8, N is the total number of people in the face image. After obtaining the corresponding speaking state label, the initial fusion feature sequence e can be obtained according to the following calculation formula n :

[0059]

[0060] Among them, e l is the first mouth feature sequence, i is the frame number corresponding to the initial mouth feature sequence, such as the first frame, the second frame, etc., n is the number of the candidate object, such as the first candidate object, the second candidate object, etc., is the speaking status of the nth candidate, The value is 1 or 0. is the total number of frames in which all candidate objects speak from the i-th frame to the T / 2-th frame.

[0061] It can be understood that, through the calculation method of formula (1), due to The value of is 1 or 0, so It can effectively distinguish the features related to speaking actions and the features related to non-speaking actions in the initial mouth feature sequence, mask the initial mouth features of non-speaking actions, and remove noise and redundant information in the features to facilitate subsequent model training.

[0062] In some embodiments, the initial fusion feature sequence and the second mouth feature sequence may be concatenated to obtain an initial concatenated feature sequence, and based on the initial concatenated feature sequence, the speech prediction probability may be obtained. Figure 3As shown, after obtaining the initial fusion feature sequence, it can be spliced ​​with the second mouth feature sequence to obtain an initial spliced ​​feature sequence with the same length as the initial mouth feature sequence. The initial spliced ​​feature sequence is input into the model prediction unit of the object detection model. After being processed by the Auto Eccoder encoding layer, the LSTM long short-term memory layer and the Sigmoid activation function layer, the speech prediction probability of the candidate object in the time window is output. For example, the speech prediction probability can be 80%, 90%, etc.

[0063] It is understandable that since the value of the speech prediction probability may have a certain error, the object detection model can be trained according to the speech prediction probability and the speech probability label of the face image sequence to obtain a trained object detection model. In some embodiments, the loss function of the trained object detection model can be calculated according to the following formula:

[0064]

[0065] in, For labels, p y is the prediction probability of speaking, and α and β are coordination parameters. When the loss function L reaches the minimum value of convergence, or the number of iterative training rounds of the object detection model reaches the target value, or the performance of the object detection model meets the requirements, the model training process is terminated to obtain a trained object detection model.

[0066] It can be understood that after the object detection model is trained in the above manner, the mouth features can be encoded through the linear output unit of the object detection model to obtain a mouth feature sequence, and the speech prediction probability of each candidate object can be obtained through the prediction of the object detection model.

[0067] It can be understood that for each candidate object in the video scene, different candidate objects may have different speaking speeds, different lengths of pauses in speech, etc., which makes the general target detection mechanism affected by the above factors and frequently switches the lens, thereby affecting the stability of speech state prediction and target detection. Therefore, by splicing the mouth feature sequence of the current time window and the historical time window, when predicting the target object, the mouth feature sequence of the historical time window can be introduced as a buffer and reference to predict the overall speech state of the candidate object within a certain period of time, thereby improving the stability of predicted speech state and target detection.

[0068] Specifically, similar to the process of obtaining the initial spliced ​​feature sequence, the mouth feature sequence of the current time window and the fusion feature sequence of the historical time window can be spliced ​​to obtain the spliced ​​feature sequence. Figure 4As shown, the mouth feature sequence of the current time window can be obtained by the output of the linear processing unit of the object detection model, and the fused feature sequence of the historical time window can be obtained by fusing the historical speech probability and the mouth feature sequence of the historical time window in the calculation method of formula (1), and then the head of the mouth feature sequence is spliced ​​to the tail of the fused feature sequence in chronological order to obtain the spliced ​​feature sequence.

[0069] It can be understood that after obtaining the spliced ​​feature sequence, since the spliced ​​feature sequence contains a fused feature sequence that integrates the speech status of historical candidate objects, the object detection model can accurately predict the target object that is speaking by combining the mouth movement status of the candidate object in the current time window and the fused features that integrate rich speech status semantics in the historical time window when predicting the target object based on the spliced ​​feature sequence. In addition, by combining the historical time window to predict the speech status in a time window of a certain length, for example, the target object is in continuous speech and is predicted in this time period, so as to avoid frequent switching of video lenses due to unstable prediction performance, thereby improving the stability of predicting and detecting the target object.

[0070] In order to better implement the target object detection method in the embodiment of the present application, the step of "detecting the target object speaking from each candidate object based on the spliced ​​feature sequence" may also include:

[0071] Performing state prediction on the concatenated feature sequence to obtain the speaking probability of each candidate object in the current time window;

[0072] According to the speaking probability of each candidate object in the current time window, the speaking target object is detected from each candidate object.

[0073] Still Figure 4 As shown, the concatenated feature sequence can be input into the model prediction unit of the object detection model to obtain the speaking probability of each candidate object in the current time window. The length of the current time window is the time length corresponding to the mouth feature sequence. As an example only, assuming that the length of the mouth feature sequence is 15 frames and that a 1-second video includes 30 frames, the speaking probability of each candidate object within the current 0.5 seconds can be predicted, for example, the speaking probability of the first candidate object is 50%, the speaking probability of the second candidate object is 60%, and so on.

[0074] In some embodiments, the specific speaking status of each candidate object can be further obtained according to the speaking probability. Specifically, an upper threshold t representing the upper limit of the speaking probability can be set first. up , and the lower limit threshold t of the lower limit of the speaking probability low , as two model parameters of the model. After obtaining the speaking probability of each candidate object, we can compare the speaking probability with the upper threshold tup , lower threshold t low The corresponding speaking state is determined by the relationship between the size of t. For example, when the speaking probability is lower than t low Indicates that the candidate is in a state of not speaking but about to speak. When the speaking probability is higher than t up Indicates that the candidate is in the state of speaking but about to finish speaking. When the speaking probability is between t low With t up Between indicates that the candidate is speaking.

[0075] In some embodiments, the speaking state of the candidate object can also be predicted by combining the changes in the speaking probabilities corresponding to the multi-stage sequence frames. As an example, assuming that the speaking probabilities of candidate object A change by 10%, 20%, and 30% in different and continuous time windows, and assuming that t low The corresponding speaking probability is 35%. It can be seen that as time goes by, the speaking probability of candidate A is increasing and approaching t low , because when the speaking probability is lower than t low Indicates that the candidate is in a state of not speaking but about to speak, and the speaking probability is between t low With t up The between indicates that the candidate object is speaking. According to the increasing speaking probability of candidate object A, it can be seen that candidate object A is more likely to speak in the next time window. Therefore, the object detection model can accurately predict the speaking status of candidate object A to improve the accuracy of detecting the target object and predicting the speaking status of the target object.

[0076] In order to better implement the target object detection method in the embodiment of the present application, after the step of "detecting the target object of the speech from each candidate object based on the spliced ​​feature sequence", the method of the present application further includes:

[0077] The speaking probability of each candidate object in the current time window and the mouth feature sequence of each candidate object in the current time window are fused to obtain a fused feature sequence of each candidate object in the current time window;

[0078] The fused feature sequence of the historical time window is updated according to the fused feature sequence of the current time window.

[0079] Still Figure 4 As shown, after obtaining the speaking probability of each candidate object in the current time window, the speaking probability can be combined with the calculation method of formula (1) in step 201 to fuse the mouth feature sequence of the candidate object in the current time window to obtain the corresponding fused feature sequence. For example, the numerator in formula (1) can be The value of is replaced with the utterance probability predicted by the object detection model.

[0080] In some embodiments, the storage queue for storing the fused feature sequence can accommodate a certain number of frames of fused feature sequences, and each newly obtained fused feature sequence can be updated to the storage queue to obtain an updated fused feature sequence.

[0081] Specifically, Figure 4 As shown, assuming that the storage queue can accommodate b-frame fusion feature sequences, each time the mouth feature sequence of the current time window of s frames can be obtained, and the mouth feature sequence of s frames can be obtained by advancing the sliding time window in a manner where the sliding step number k is 1. Assuming that the initial length of the fusion feature sequence stored in the storage queue is 15 frames, and the value of s is 15, then the mouth feature sequence of the current time window of 15 frames is obtained by advancing one step according to the sliding step number k, and then the speech probability of each candidate object in the current time window is fused to obtain the current fusion feature sequence of 15 frames in length, and then the current fusion feature sequence is spliced ​​with the stored fusion feature sequence to obtain a fusion feature sequence of 30 frames in length as the updated fusion feature sequence. Assuming that the storage upper limit of the storage queue is 100 frames, the length of the fusion feature sequence is updated to 100 frames in the above manner, and then the above update process is stopped, and the speaking status of the candidate object is predicted based on the fusion feature sequence under the b-frame length.

[0082] In order to better implement the target object detection method in the embodiment of the present application, the step of fusing the speaking probability of each candidate object in the current time window and the mouth feature sequence of each candidate object in the current time window to obtain the fused feature sequence of each candidate object in the current time window includes:

[0083] Determine the first speaking frame number of each candidate object in the current time window according to the speaking probability of each candidate object in the current time window;

[0084] Determining the second speaking frame number of each candidate object in the historical time window according to the speaking probability of each candidate object in the historical time window;

[0085] Determine the total number of speech frames of each candidate object according to the first number of speech frames and the second number of speech frames;

[0086] Based on the total number of speech frames, the speech probability of each candidate object in the current time window and the mouth feature sequence of each candidate object in the current time window are fused to obtain the fused feature sequence.

[0087] According to the previous text, the is the total number of frames in which all candidate objects are speaking from the i-th frame to the T / 2-th frame, and the T / 2 frame in formula (2) includes the frames corresponding to the historical time window and the current time window of all candidate objects. Therefore, as a necessary condition for mouth feature fusion, it is necessary to calculate the first speaking frame number of each candidate object in the current time window and the second speaking frame number in the historical time window, and sum the first speaking frame number and the second speaking frame number to obtain the total speaking frame number corresponding to each candidate object.

[0088] In some embodiments, the total number of speech frames in the corresponding time window can be calculated based on the speech probability. For example, the speech probability of candidate B in the current time window is 60%. low With t up , indicating that the candidate B is speaking, the corresponding total number of speech frames is the total number of frames of the current time window length, for example, it can be 15 frames, 20 frames, 30 frames, etc. Correspondingly, if the speech probability under the current time window indicates that the candidate is in a state of not speaking, the corresponding total number of speech frames can be recorded as 0 frames.

[0089] It should be noted that since the length of each time window is set according to the prediction sensitivity requirement, and the sensitivity is usually set high, taking a 1-second video including 30 frames as an example, if the length of each time window is set to 15 frames, each time window corresponds to 0.5 seconds, and if the length of each time window is set to 1 frame, each time window corresponds to 1 / 30 seconds. It can be understood that since the duration of each target object's speech in the video scene is generally greater than 10 seconds, the number of frames contained in each time window will not occupy a large proportion of the total number of speech frames. Therefore, even if the number of frames corresponding to the length of the time window is all recorded as the number of speech frames in the above manner, or none of them are recorded as the number of speech frames, it will not affect the accuracy of the fused mouth feature sequence and the subsequent prediction of the target object.

[0090] In order to better implement the target object detection method in the embodiment of the present application, after the step of "detecting the target object of the speech from each candidate object based on the spliced ​​feature sequence", the method of the present application further includes:

[0091] The speech start time and speech end time of the target object are determined according to the speech probability of the target object in the historical time window and the speech probability in the current time window.

[0092] As mentioned above, if the candidate's speaking probability in multiple consecutive time windows is constantly increasing or instantaneously increasing, and is lower than and close to the lower limit threshold t low, the candidate object is more likely to speak in the next time window, so that the object detection model can accurately predict the start time of the candidate object's speech in this way. Correspondingly, if the candidate object is in the state of speaking, and the probability of speaking in multiple consecutive time windows is constantly increasing or instantly rising to above the upper threshold t up , it means that the candidate object is more likely to finish speaking in the next time window, so that the object detection model can accurately predict the end time of the candidate object's speech in this way.

[0093] It can be seen that the target object detection method, device, terminal device and computer-readable storage medium provided by the present application process the facial image of the candidate object, extract relatively rich feature information of the candidate object, detect the speaking target object based on the rich feature information, and predict the speaking probability, speaking status and speaking timing of each candidate object based on the spliced ​​feature sequence, thereby effectively improving the accuracy and stability of detecting the speaking target object in the video scene.

[0094] In order to better implement the target object detection method in the embodiment of the present application, after the step of "detecting the target object of the speech from each candidate object based on the spliced ​​feature sequence", the method of the present application further includes:

[0095] Acquire a first video frame displaying the target object, and a second video frame adjacent to the first video frame, wherein the first video frame displays a first detection frame, and the second video frame displays a second detection frame;

[0096] Determine the facial similarity between the object in the second detection frame and the target object according to the aspect ratio of the first detection frame, the aspect ratio of the second detection frame, the intersection-over-union ratio of the first lock frame and the second detection frame, the center distance between the first detection frame and the second detection frame, and the longest diagonal length of the first detection frame and the second detection frame;

[0097] Based on the face similarity, a detection status of the object in the second detection frame is determined.

[0098] In the video scene, the speaker will be detected, tracked, and close-up to ensure that the speaker is in the center of the video screen as much as possible. When the speaker's position changes, the speaker's area in the video screen may also change, causing the position and size of the detection frame used to identify and detect the speaker to change accordingly. The similarity of the speaker's position in the video screen is usually measured by calculating the intersection and union ratio between the two detection frames of two adjacent frames. However, since the above changes in the position and size of the speaker in the video will produce a large lens drift, the accuracy of continuously detecting the speaker through position similarity will be greatly reduced.

[0099] In some embodiments, the face similarity between the object in the first detection frame a and the object in the second detection frame b can be calculated based on the following formula:

[0100]

[0101]

[0102] γ=(r a -r b ) 2 +σ (6)

[0103] Among them, r a is the aspect ratio of the first detection frame, r b is the aspect ratio of the second detection box, σ is a hyperparameter (usually 0.000001), d center is the center distance between the first detection frame and the second detection frame, d dig is the longest diagonal length of the first detection frame and the second detection frame, and α is the intersection-over-union ratio of the first detection frame to the second detection frame.

[0104] When the target object is in the state of speaking, the face similarity of the object in the two adjacent frames can be calculated according to formula (4) to search and find the corresponding target object. Then, the area of ​​the video screen where the target object is located is cut out and displayed in the center of the video screen to achieve detection, tracking and close-up of the target object.

[0105] In order to better implement the target object detection method in the embodiment of the present application, after the step of "detecting the target object of the speech from each candidate object based on the spliced ​​feature sequence", the method of the present application further includes:

[0106] Obtaining the offset of the target object relative to the center of the video screen;

[0107] Based on the offset, a pan / tilt angle corresponding to the pan / tilt used to shoot the video image is determined.

[0108] Optionally, the step of “determining a pan / tilt rotation angle corresponding to the pan / tilt for shooting the video image based on the offset” may include:

[0109] Calculate the initial PTZ conversion coefficient;

[0110] Acquire a target object that meets the picture adjustment conditions, detect the face of the target object, and calculate the offset of the face of the target object from the frontal angle relative to the center of the picture;

[0111] Determine whether the face of the target object at the current angle is in the center of the picture, if so, determine the initial pan-tilt conversion coefficient as the target pan-tilt conversion coefficient, if not, dynamically update the initial pan-tilt conversion coefficient until the target pan-tilt conversion coefficient that meets the conditions is obtained;

[0112] The gimbal rotation angle is calculated based on the offset and the target gimbal conversion coefficient.

[0113] In video scenes, the movement of the target object during the speech process will make the face of the person turn sideways to the gimbal, which seriously affects the close-up effect of the target object's face. In order to obtain a good close-up effect, it is necessary to estimate the angle of the target object's side face and guide the gimbal to rotate to achieve the best shooting angle. Among them, assuming that the maximum rotation angle of the gimbal is 180 degrees, the conversion relationship between the gimbal rotation angle and the target object's face offset is:

[0114] R = k ptz d (7)

[0115] Where R is the horizontal and vertical offset angle of the gimbal, k ptz is the pan / tilt conversion coefficient, and d is the offset of the target object’s face center relative to the screen center.

[0116] In some embodiments, the gimbal conversion coefficient may be calculated by the following steps:

[0117] S1. Calculate the initial PTZ conversion coefficient:

[0118]

[0119] Among them, fov is the field of view of the gimbal lens, and w is half of the maximum resolution of the lens;

[0120] S2. Acquire a target object that meets the picture adjustment conditions.

[0121] In some embodiments, the image adjustment condition may be that the image captured by the gimbal only contains the face of the target object, and the face of the target object is not facing the lens of the gimbal.

[0122] S3. Detect the face of the target object and calculate the offset d of the front angle of the face relative to the center of the picture.

[0123] S4, judging whether the face of the target object at the current angle is in the center of the picture, if so, determining the initial pan-tilt conversion coefficient as the target pan-tilt conversion coefficient, if not, updating the initial pan-tilt conversion coefficient according to the positional relationship between the face center and the picture center, and obtaining an updated pan-tilt conversion coefficient, that is:

[0124] k ptz = k ptz + τδ (9)

[0125] Among them, τ is the sign function, the face exceeds the center of the picture and takes -1, otherwise it takes 1, and δ is the step length coefficient, which is calculated by the following formula:

[0126]

[0127] in, is the maximum step size parameter;

[0128] S5. Calculate the offset angle of the gimbal according to the offset and the updated gimbal conversion coefficient, rotate the gimbal, and jump to step S2.

[0129] In order to better implement the target object detection method in the embodiment of the present application, in addition to the target object detection method, the embodiment of the present application also provides a face detection model training device, such as Figure 6 As shown, the device comprises:

[0130] The face image processing module 301 is used to process the face image of each candidate object to obtain feature information of each candidate object;

[0131] The target object detection module 302 is used to process the feature information and detect the target object of the speech from each candidate object.

[0132] The target object detection module 302 of the present application provides a face detection model training device, which processes the face image of each candidate object through the face image processing module 301 to obtain feature information of each candidate object; and processes the feature information through the target object detection module 302 to detect the target object of the speech from each candidate object.

[0133] In some embodiments of the present application, the target object detection module 302 is further specifically configured to:

[0134] Performing state prediction on the concatenated feature sequence to obtain the speaking probability of each candidate object in the current time window;

[0135] According to the speaking probability of each candidate object in the current time window, the speaking target object is detected from each candidate object.

[0136] In some embodiments of the present application, the target object detection module 302 is further specifically used for:

[0137] Performing state prediction on the concatenated feature sequence to obtain the speaking probability of each candidate object in the current time window;

[0138] According to the speaking probability of each candidate object in the current time window, the speaking target object is detected from each candidate object.

[0139] In some embodiments of the present application, the device further includes:

[0140] A current fusion module, used to fuse the speaking probability of each candidate object in the current time window and the mouth feature sequence of each candidate object in the current time window to obtain a fused feature sequence of each candidate object in the current time window;

[0141] The fusion feature updating module is used to update the fusion feature sequence of the historical time window according to the fusion feature sequence of the current time window.

[0142] In some embodiments of the present application, the current fusion module is further specifically used for:

[0143] Determine the first speaking frame number of each candidate object in the current time window according to the speaking probability of each candidate object in the current time window;

[0144] Determining the second speaking frame number of each candidate object in the historical time window according to the speaking probability of each candidate object in the historical time window;

[0145] Determine the total number of speech frames of each candidate object according to the first number of speech frames and the second number of speech frames;

[0146] Based on the total number of speech frames, the speech probability of each candidate object in the current time window and the mouth feature sequence of each candidate object in the current time window are fused to obtain the fused feature sequence.

[0147] In some embodiments of the present application, the device further includes:

[0148] The speaking time determination module is used to determine the speaking start time and speaking end time of the target object according to the speaking probability of the target object in the historical time window and the speaking probability in the current time window.

[0149] In some embodiments of the present application, the device further includes:

[0150] An adjacent video frame acquisition module, used to acquire a first video frame displaying the target object, and a second video frame adjacent to the first video frame, wherein the first video frame displays a first detection frame, and the second video frame displays a second detection frame;

[0151] a face similarity acquisition module, configured to determine a face similarity between an object in the second detection frame and the target object according to an aspect ratio of the first detection frame, an aspect ratio of the second detection frame, an intersection-over-union ratio of the first detection frame and the second detection frame, a center distance between the first detection frame and the second detection frame, and a length of the longest diagonal line between the first detection frame and the second detection frame;

[0152] The object locking module is used to lock the object in the second detection frame if the face similarity is greater than a set threshold.

[0153] In some embodiments of the present application, the device further includes:

[0154] An initial PTZ conversion coefficient calculation module is used to calculate the initial PTZ conversion coefficient;

[0155] An offset calculation module is used to obtain a target object that meets the picture adjustment condition, detect the face of the target object, and calculate the offset of the front angle of the face of the target object relative to the center of the picture;

[0156] A face angle determination module, used to determine whether the current position of the face of the target object at the current angle is in the center of the picture, if so, determine the initial pan-tilt conversion coefficient as the target pan-tilt conversion coefficient, if not, dynamically update the initial pan-tilt conversion coefficient until the target pan-tilt conversion coefficient that meets the conditions is obtained;

[0157] The gimbal angle calculation module is used to calculate the gimbal angle based on the offset and the target gimbal conversion coefficient.

[0158] The present application also provides a terminal device, which includes a processor, a memory, and a computer program stored in the memory and executable on the processor, and the processor executes the computer program to implement the steps of any target object detection method in the present application. The terminal device integrates any target object detection method provided in the present application. Figure 7 As shown, it shows a schematic diagram of the structure of the terminal device involved in the embodiment of the present application, specifically:

[0159] The terminal device may include one or more processing core processors 401, one or more computer-readable storage media memories 402, a power supply 403, an input unit 404 and other components. Those skilled in the art will appreciate that Figure 7 The terminal device structure shown in the figure does not constitute a limitation on the terminal device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. Among them:

[0160] The processor 401 is the control center of the terminal device, and uses various interfaces and lines to connect various parts of the entire terminal device. By running or executing software programs and / or modules stored in the memory 402, and calling data stored in the memory 402, the processor 401 executes various functions of the terminal device and processes data, thereby monitoring the terminal device as a whole. Optionally, the processor 401 may include one or more processing cores; the processor 401 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. Preferably, the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, the user interface and the application program, etc., and the modem processor mainly processes wireless communication. It is understandable that the above-mentioned modem processor may not be integrated into the processor 401.

[0161] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the terminal device, etc. In addition, the memory 402 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0162] The terminal device also includes a power supply 403 for supplying power to each component. Preferably, the power supply 403 can be logically connected to the processor 401 through a power management system, so as to manage charging, discharging, power consumption management and other functions through the power management system. The power supply 403 can also include any components such as one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators.

[0163] The terminal device may further include an input unit 404, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0164] Although not shown, the terminal device may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 401 in the terminal device will load the executable files corresponding to the processes of one or more application programs into the memory 402 according to the following instructions, and the processor 401 will run the application programs stored in the memory 402, thereby realizing various functions, such as:

[0165] Get the image to be processed;

[0166] Determine a first area and a second area of ​​the image to be processed, wherein the first area and the second area are non-overlapping image areas;

[0167] The first region and the second region are enhanced in sequence to obtain a target image, wherein the degree of change of pixel values ​​in the first region of the target image is smaller than the degree of change of pixel values ​​in the second region of the target image.

[0168] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0169] To this end, the embodiment of the present application provides a computer-readable storage medium, which may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc. A computer program is stored thereon, and the computer program is loaded by a processor to execute the steps in any target object detection method provided in the embodiment of the present application. For example, the computer program can be loaded by a processor to execute the following steps:

[0170] Get the image to be processed;

[0171] Determine a first area and a second area of ​​the image to be processed, wherein the first area and the second area are non-overlapping image areas;

[0172] The first region and the second region are enhanced in sequence to obtain a target image, wherein the degree of change of pixel values ​​in the first region of the target image is smaller than the degree of change of pixel values ​​in the second region of the target image.

[0173] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the detailed description of other embodiments above, and will not be repeated here.

[0174] In specific implementation, the above units or structures can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above units or structures can refer to the previous method embodiments, which will not be repeated here.

[0175] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0176] The above is a detailed introduction to a target object detection method and device provided in an embodiment of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A target object detection method, characterized in that: The method comprises: Processing the facial image of each candidate object to obtain feature information of each candidate object; The feature information is processed to detect the speaking target object from each candidate object.

2. The method according to claim 1, characterized in that The feature information includes a mouth feature sequence of each candidate object in the current time window; The processing of the feature information to detect the target object of the speech from each candidate object includes: Acquire a fused feature sequence of a historical time window, wherein the fused feature sequence represents a feature sequence of a speech state of each candidate object; splicing the mouth feature sequence and the fusion feature sequence to obtain a spliced ​​feature sequence; Based on the concatenated feature sequence, the target object of the speech is detected from each candidate object.

3. The method according to claim 2, characterized in that The step of detecting the target object of the speech from each candidate object based on the concatenated feature sequence comprises: Performing state prediction on the concatenated feature sequence to obtain the speaking probability of each candidate object in the current time window; According to the speaking probability of each candidate object in the current time window, the speaking target object is detected from each candidate object.

4. The method according to claim 2, characterized in that: After detecting the target object of the speech from each candidate object based on the spliced ​​feature sequence, the method further includes: The speaking probability of each candidate object in the current time window and the mouth feature sequence of each candidate object in the current time window are fused to obtain a fused feature sequence of each candidate object in the current time window; The fused feature sequence of the historical time window is updated according to the fused feature sequence of the current time window.

5. The method according to claim 4, characterized in that The speaking probability of each candidate object in the current time window and the mouth feature sequence of each candidate object in the current time window are fused to obtain a fused feature sequence of each candidate object in the current time window, including: Determine the first speaking frame number of each candidate object in the current time window according to the speaking probability of each candidate object in the current time window; Determining the second speaking frame number of each candidate object in the historical time window according to the speaking probability of each candidate object in the historical time window; Determine the total number of speech frames of each candidate object according to the first number of speech frames and the second number of speech frames; Based on the total number of speech frames, the speech probability of each candidate object in the current time window and the mouth feature sequence of each candidate object in the current time window are fused to obtain the fused feature sequence.

6. The method according to claim 5, characterized in that After detecting the target object of the speech from each candidate object, the method further includes: The speech start time and speech end time of the target object are determined according to the speech probability of the target object in the historical time window and the speech probability in the current time window.

7. The method according to claim 1, characterized in that After detecting the target object speaking from each candidate object, the method further includes: Acquire a first video frame displaying the target object, and a second video frame adjacent to the first video frame, wherein the first video frame displays a first detection frame, and the second video frame displays a second detection frame; Determine the facial similarity between the object in the second detection frame and the target object according to the aspect ratio of the first detection frame, the aspect ratio of the second detection frame, the intersection-over-union ratio of the first lock frame and the second detection frame, the center distance between the first detection frame and the second detection frame, and the longest diagonal length of the first detection frame and the second detection frame; If the face similarity is greater than a set threshold, the object in the second detection frame is locked.

8. The method according to claim 1, characterized in that After detecting the target object speaking from each candidate object, the method further includes: Obtaining the offset of the target object relative to the center of the video screen; Based on the offset, a pan / tilt angle corresponding to the pan / tilt used to shoot the video image is determined.

9. The method according to claim 8, characterized in that Determining a pan / tilt angle corresponding to a pan / tilt for shooting a video image based on the offset includes: Calculate the initial PTZ conversion coefficient; Acquire a target object that meets the picture adjustment conditions, detect the face of the target object, and calculate the offset of the face of the target object from the frontal angle relative to the center of the picture; Determine whether the face of the target object at the current angle is in the center of the picture, if so, determine the initial pan-tilt conversion coefficient as the target pan-tilt conversion coefficient, if not, dynamically update the initial pan-tilt conversion coefficient until the target pan-tilt conversion coefficient that meets the conditions is obtained; The gimbal rotation angle is calculated based on the offset and the target gimbal conversion coefficient.

10. A target object detection device, characterized in that: The device comprises: A face image processing module is used to process the face image of each candidate object to obtain feature information of each candidate object; The target object detection module is used to process the feature information and detect the target object of the speech from each candidate object.

11. A terminal device, characterized in that: The terminal device includes a processor, a memory, and a computer program stored in the memory and executable on the processor, and the processor executes the computer program to implement the steps in the target object detection method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps in the target object detection method according to any one of claims 1 to 9.

Citation Information

Cited By

  • Vision-based access control method, storage medium and access control equipment

    CN122157404A