Video processing method, device and equipment for sound source object detection

Abstract

The invention discloses a video processing method, device and equipment for sound source object detection, and the method comprises the steps: determining target recognition result information based on object recognition information and video information;

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a video processing method, device and equipment for sound source object detection. Background Art

[0002] 3D sound effects can meet consumers' demand for live sound effects by creating a sense of surround sound in the front, back, left, right, and especially in the vertical direction. One of the core technologies of 3D sound effects is sound source object detection. Existing sound source object detection methods can generally only process offline video sources and cannot provide universal and real-time surround sound services. Summary of the invention

[0003] Embodiments of the present application provide a video processing method, apparatus, and device for sound source object detection.

[0004] In a first aspect, the present application provides a video processing method, comprising:

[0005] Get video information;

[0006] Determining object recognition information based on the video information;

[0007] Based on the object recognition information and the video information, target recognition result information is determined.

[0008] In a second aspect, the present application provides a video processing device, including:

[0009] An information acquisition module, used for acquiring video information;

[0010] An object recognition module, used to determine object recognition information based on video information;

[0011] The video processing module is used to determine target recognition result information based on the object recognition information and the video information.

[0012] In a third aspect, the present application further provides a computer device, the computer device comprising:

[0013] one or more processors;

[0014] Memory; and

[0015] One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the processor to implement any one of the video processing methods in the first aspect.

[0016] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, and the computer program is loaded by a processor to execute the steps in any one of the video processing methods in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 is a schematic diagram of a scenario of a video processing system provided by an embodiment of the present invention;

[0019] Figure 2 is a flow chart of an embodiment of a video processing method provided by an embodiment of the present invention;

[0020] Figure 3 is a flowchart of a specific embodiment of obtaining video information provided by an embodiment of the present invention;

[0021] Figure 4 is a flowchart of a specific embodiment of determining object identification information provided by an embodiment of the present invention;

[0022] Figure 5 is a flowchart of a specific embodiment of determining target recognition result information provided by an embodiment of the present invention;

[0023] Figure 6 is a schematic diagram of a specific embodiment of object feature point information of a target object provided by an embodiment of the present invention;

[0024] Figure 7 is a flow chart of another embodiment of the video processing method provided by an embodiment of the present invention;

[0025] Figure 8 is a principle block diagram of a video processing device provided by an embodiment of the present invention;

[0026] Fig. 9 It is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0028] In the description of the present application, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present application. In addition, the terms "first", "second", "third" and the like are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, features defined as "first", "second", "third" and the like may explicitly or implicitly include one or more of the said features.

[0029] In this application, the word "exemplary" is used to mean "used as an example, illustration, or description." Any embodiment described in this application as "exemplary" is not necessarily to be construed as being preferred or advantageous over other embodiments. The following description is given to enable any technician in the field to implement and use the present application. In the following description, details are listed for the purpose of explanation. It should be understood that a person of ordinary skill in the art can recognize that the present application can be implemented without using these specific details. In other instances, well-known structures and processes will not be elaborated in detail to avoid obscuring the description of the present application with unnecessary details. Therefore, the present application is not intended to be limited to the embodiments shown, but is consistent with the widest scope consistent with the principles and features disclosed in the present application.

[0030] It should be noted that since the method of the embodiment of the present application is executed in a computer device, the processing objects of each computer device exist in the form of data or information. For example, time is actually time information. It can be understood that if size, quantity, position, etc. are mentioned in subsequent embodiments, they are all corresponding data for processing by the computer device. The details will not be repeated here.

[0031] The embodiments of the present application provide a video processing method, apparatus and device for sound source object detection, which are described in detail below.

[0032] See also Figure 1 , Figure 1 Schematic diagram of a video processing system provided in an embodiment of the present application. The video processing system may include a computer device 100, in which a video processing device is integrated. Figure 1 Computer equipment in.

[0033] In the embodiment of the present application, the computer device 100 is mainly used to obtain video information; based on the video information, determine the object recognition information; based on the object recognition information and the video information, determine the target recognition result information, and can perform real-time sound source object detection, thereby providing real-time surround sound service.

[0034] In the embodiment of the present application, the computer device 100 may be an independent server, or a server network or server cluster composed of servers. For example, the computer device 100 described in the embodiment of the present application includes but is not limited to a computer, a network host, a single network server, a plurality of network server sets or a cloud server composed of a plurality of servers. The cloud server is composed of a large number of computers or network servers based on cloud computing.

[0035] It is understandable that the computer device 100 used in the embodiments of the present application may be a device including both receiving and transmitting hardware, that is, a device having receiving and transmitting hardware capable of performing two-way communication on a two-way communication link. Such a device may include: a cellular or other communication device having a single-line display or a multi-line display or a cellular or other communication device without a multi-line display. The specific computer device 100 may be a desktop terminal or a mobile terminal, and the computer device 100 may also be one of a mobile phone, a tablet computer, a laptop computer, etc.

[0036] Those skilled in the art will understand that Figure 1 The application environment shown in the figure is only one application scenario of the present application solution and does not constitute a limitation on the application scenario of the present application solution. Other application environments may also include Figure 1 More or less computer equipment as shown in Figure 1 Only one computer device is shown in the figure. It can be understood that the video processing system can also include one or more other services, which are not specifically limited here.

[0037] In addition, if Figure 1 As shown, the video processing system may also include a memory 200 for storing data, such as object feature point information, such as a first object feature point, a second object feature point, a third object feature point, etc., such as video information, such as a first target object information, object feature point information, etc.

[0038] It should be noted that Figure 1The scene diagram of the video processing system shown is only an example. The video processing system and scene described in the embodiment of the present application are intended to more clearly illustrate the technical solution of the embodiment of the present application, and do not constitute a limitation on the technical solution provided in the embodiment of the present application. Ordinary technicians in this field can know that with the evolution of the video processing system and the emergence of new business scenarios, the technical solution provided in the embodiment of the present application is also applicable to similar technical problems.

[0039] First, an embodiment of the present application provides a video processing method, device and equipment for sound source object detection. The executor of the video processing method for sound source object detection is a video processing device, and the video processing device is applied to a computer device. The video processing method includes: obtaining video information; determining object recognition information based on the video information; and determining target recognition result information based on the object recognition information and the video information.

[0040] like Figure 2 FIG. 2 is a flow chart of an embodiment of a video processing method in an embodiment of the present application. The video processing method may include the following steps S201 to S203, which are as follows:

[0041] S201. Obtain video information.

[0042] In this embodiment, the video information is information extracted from the first video frame information, the first video frame information is video image information that currently needs to perform sound source object detection, the first video frame information includes a target object, the target object is a person or object that can make sound, the target object includes but is not limited to common sound-making objects such as vehicles, airplanes, ships, people, birds, etc., and the video information can be feature information of the target object extracted from the first video frame information.

[0043] In a specific embodiment, the video information includes first target object information and object feature point information, the first target object information represents the position information of the target object in the first video frame information, and the object feature point information represents the key point information of the target object in the first video frame information. For example, the first video frame information includes person A and person B, the first target object information includes bounding box information of person A and person B in the first video frame information, and the object feature point information includes facial key point information of person A and person B in the first video frame information.

[0044] Accordingly, refer to Figure 3 As shown, the acquisition of video information in the above step S201 may include steps S301 to S303, which are specifically as follows:

[0045] S301. Obtain first video frame information.

[0046] In a specific embodiment, the step of obtaining the first video frame information specifically includes: obtaining video information; performing video frame extraction on the video information to obtain the first video frame information. The video information may be video information obtained by an imaging module configured by the computer device itself, or may be video information shot by a high-definition camera, or may be video information acquired by an imaging module of other computer devices through the network, Bluetooth, infrared, etc., and the embodiment of the present invention does not limit this. For example, when the video processing method of the present application is applied to a smart phone, the smart phone can directly obtain video information through the imaging module configured by itself. When the data processing method of the present application is applied to a server, the server can collect video information through the imaging module of the smart phone, and obtain the video information from the smart phone through the network, Bluetooth, infrared, etc.

[0047] S302: Perform target detection on the first video frame information to obtain first target object information.

[0048] In a specific embodiment, the first target object information is the bounding box information of the target object obtained by performing target detection on the first video frame information. The first target object information can represent the position information of the target object in the first video frame information. For example, if the target object is a person, the first target object information is the face detection box information of the person.

[0049] After obtaining the first video frame information, this embodiment can perform target detection on the first video frame information to obtain first target object information. The target detection on the first video frame information can adopt an existing target detection algorithm, such as a traditional target detection algorithm, a target detection algorithm based on deep learning, etc. The specific details can be referred to the existing technology, and this application will not go into details here.

[0050] S303: Perform feature point detection on the first video frame information based on the first target object information to obtain object feature point information.

[0051] The object feature point information is key point information of the target object obtained by performing feature point detection on the first video frame information based on the first target object information. For example, the target object is a person, and the first target object information is face detection frame information of the target object. Feature point detection is performed on the first video frame information based on the face detection frame information to obtain face key point information of the target object.

[0052] S202: Determine object recognition information based on the video information.

[0053] The object identification information is used to uniquely identify the target object. For example, the object identification information may be an ID number, a user name, etc. of the target object, which is not limited in this embodiment. In a specific implementation, the video information includes the first target object information. Figure 4As shown, the above step S202 determines the object recognition information based on the video information, which may include the following steps S401 to S402, which are specifically as follows:

[0054] S401: Obtain a video frame type of first video frame information.

[0055] In this embodiment, the video frame type includes a first type and / or a second type, the first type indicates that the first video frame information is the first frame video frame information, and the second type indicates that the first video frame information is non-first frame video frame information.

[0056] S402: Determine object recognition information based on the video frame type and the first target object information.

[0057] In a specific embodiment, when the video frame type is the first type, the step of determining the object recognition information based on the video frame type and the first target object information specifically includes: determining the object recognition information based on the first target object information.

[0058] In another specific embodiment, when the video frame type is the second type, the step of determining the object recognition information based on the video frame type and the first target object information specifically includes: determining the object recognition information based on the first target object information and the historical video information. That is, if the first video frame information is the first frame video frame information, the object recognition information is determined based on the first target object information; if the first video frame information is not the first frame video frame information, the object recognition information is determined based on the first target object information and the historical video information.

[0059] For example, if the video frame type is the first type, that is, the first video frame information is the first frame video frame information, if the first target object information includes face frame A, face frame B and face frame C, the object recognition information is determined as face frame A corresponding to ID number 001, face frame B corresponding to ID number 002, and face frame C corresponding to ID number 003.

[0060] In a specific embodiment, the historical video information includes several historical video frame information. Based on the first target object information and the historical video information, the step of determining the object recognition information specifically includes: determining the similarity information based on the first target object information and the historical video frame information in the historical video information; and determining the object recognition information based on the similarity information.

[0061] Optionally, based on the first target object information and the historical video frame information in the historical video information, the step of determining the similarity information specifically includes: performing a first fusion process on the first target object information and the second target object information of the historical video frame information to obtain third target object information; performing a second fusion process on the first target object information and the second target object information to obtain fourth target object information; performing a division process on the third target object information and the fourth target object information to obtain similarity information.

[0062] In a specific embodiment, the first target object information is the face detection frame of the target object in the first video frame information, the second target object information is the face detection frame of the target object in the historical video frame information, the first fusion processing refers to performing an intersection operation on the face detection frame of the target object in the first video frame information and the face detection frame of the target object in the historical video frame information, and the second fusion processing refers to performing a union operation on the face detection frame of the target object in the first video frame information and the face detection frame of the target object in the historical video frame information. Correspondingly, the similarity information is the intersection-over-union ratio of the face detection frame of the target object in the first video frame information and the face detection frame of the target object in the historical video frame information. Among them, the determination process of the similarity information can be expressed as: IOU = A∩B / A∪B, IOU represents the similarity information, A represents the first target object information, and B represents the second target object information.

[0063] Optionally, the similarity information includes several similarities, and the above-mentioned step of determining the object recognition information based on the similarity information specifically includes: determining the target similarity from several similarities based on a preset similarity threshold, and determining the object recognition information based on the target similarity; if the target similarity cannot be determined from several similarities based on the preset similarity threshold, determining the object recognition information based on the object information of the historical video frame information. Among them, the target similarity represents the similarity greater than the similarity threshold, and the meaning of the representation is to represent or is, that is, the target similarity represents the similarity greater than the similarity threshold or the target similarity is the similarity greater than the similarity threshold.

[0064] For example, if the similarity between the face frame A in the first video frame information and the face frame B in the historical video frame information is greater than the similarity threshold, the ID number corresponding to the face frame B is determined as the object recognition information of the face frame A; if the similarity between the face frame A in the first video frame information and the face frame B, face detection frame C, and face detection frame D in the historical video frame information are all less than or equal to the similarity threshold, and the ID numbers corresponding to the face frame B, face detection frame C, and face detection frame D in the historical video frame information are 001, 002, and 003 respectively, then 004 is determined as the object recognition information of the face frame A.

[0065] S203: Determine target recognition result information based on the object recognition information and the video information.

[0066] The target recognition result information includes whether the target object in the first video frame information makes a sound, and / or whether the sound-making object switches. This embodiment determines the object recognition information based on the video information, and then determines the target recognition result information based on the object recognition information and the video information. Real-time sound source object detection can be performed, thereby providing real-time surround sound services.

[0067] In a specific implementation, the video information includes object feature point information, referring to Figure 5 As shown, in the above step S203, based on the object recognition information and the video information, the target recognition result information is determined, which may include the following steps S501 to S503, which are specifically as follows:

[0068] S501. Determine first object feature information based on object feature point information.

[0069] In a specific embodiment, the step of determining the first object feature information based on the object feature point information specifically includes: analyzing and processing the object feature point information to obtain analysis feature value information; analyzing and processing the analysis feature value in the analysis feature value information to obtain the first object feature information.

[0070] The analysis characteristic value information includes M analysis characteristic values, where M is a positive integer not less than 2. For example, Figure 6 As shown, the object feature point information is the key point information of the face, and the analyzed feature value information includes the height of the mouth, the height of the mouth opening, the distance from the nose to the upper lip, etc.

[0071] In a specific embodiment, the analysis feature value includes a first analysis feature value, and the first analysis feature value represents the height of the first target part. For example, the first target part is the mouth, and the first analysis feature value represents the height of the mouth, that is, the distance from the upper edge of the upper lip to the lower edge of the lower lip. The first analysis feature value is obtained by analyzing and processing the object feature point information based on the first feature analysis model, and the object feature point information includes the first object feature point, the second object feature point, the third object feature point, the fourth object feature point, the fifth object feature point and the sixth object feature point of the first target part. The first feature analysis model is: S 1 =γ 1 *d 1 +γ 2 *d 2 +γ 3 *d 3 , S 1 represents the first analytical eigenvalue, d 1 represents the distance between the first object feature point and the second object feature point, d2 represents the distance between the feature point of the third object and the feature point of the fourth object, d 3 represents the distance between the fifth object feature point and the sixth object feature point, γ 1 , γ 2 and γ 3 Indicates preset parameters.

[0072] Optionally, γ 1 , γ 2 and γ 3 It can be set according to actual needs. In a specific embodiment, 0<γ 1 ≤1,0<γ 2 ≤1,0<γ 3 ≤1, preferably, γ 1 =1,γ 2 =1,γ 3 =1, then S 1 =d 1 +d 2 +d 3 .

[0073] For example, refer to Figure 6 As shown, the object feature point information includes the first object feature point 51, the second object feature point 59, the third object feature point 52, the fourth object feature point 58, the fifth object feature point 53 and the sixth object feature point 57 of the mouth, and the first feature analysis model is: mouth_height = d(lmks

[51] , lmks

[59] )+d(lmks

[52] , lmks

[58] )+d(lmks

[53] , lmks

[57] ), mouth_height represents the first analysis feature value, d(limks

[51] , lmks

[59] ) represents the distance between the first object feature point 51 and the second object feature point 59, d(lmks

[52] , lmks

[58] ) represents the distance between the third object feature point 52 and the fourth object feature point 58, and d(lmks

[53] , limks

[57] ) represents the distance between the fifth object feature point 53 and the sixth object feature point 57.

[0074] In a specific embodiment, the analysis feature value also includes a second analysis feature value, the second analysis feature value represents the opening and closing degree of the first target part, and the opening and closing degree represents the degree to which the first target part changes from a closed or contracted state to an open or expanded state. For example, the first target part is the mouth, and the opening and closing degree represents the opening height of the mouth, that is, the distance from the lower edge of the upper lip to the upper edge of the lower lip. The second analysis feature value is obtained by analyzing and processing the object feature point information based on the second feature analysis model, and the object feature point information also includes the seventh object feature point, the eighth object feature point, the ninth object feature point, the tenth object feature point, the eleventh object feature point and the twelfth object feature point of the first target part. The second feature analysis model is: S 2 =γ 4 *d 4 +γ 5 *d 5 +γ 6 *d 6 , S 2 represents the second analytical eigenvalue, d 4 represents the distance between the seventh object feature point and the eighth object feature point, d 5 represents the distance between the ninth object feature point and the tenth object feature point, d 6 represents the distance between the feature point of the eleventh object and the feature point of the twelfth object, γ 4 , γ 5 and γ 6 Indicates preset parameters.

[0075] Optionally, γ 4 , γ 5 and γ 6 It can be set according to actual needs. In a specific embodiment, 0<γ 4 ≤1,0<γ 5 ≤1,0<γ 6 ≤1, preferably, γ 4 =1,γ 5 =1,γ 6 =1, then S 2 =d 4 +d 5 +d 6 .

[0076] For example, continue to refer to Figure 6As shown, the object feature point information also includes the seventh object feature point 62, the eighth object feature point 68, the ninth object feature point 63, the tenth object feature point 67, the eleventh object feature point 64 and the twelfth object feature point 66 of the mouth, and the second feature analysis model is: openness = d(lmks

[62] , lmks

[68] ) + d(lmks

[63] , lmks

[67] ) + d(lmks

[64] , lmks

[66] ), wherein openness represents the second analysis feature value, d(limks

[62] , lmks

[68] ) represents the distance between the seventh object feature point 62 and the eighth object feature point 68, d(lmks

[63] , lmks

[67] ) represents the distance between the ninth object feature point 63 and the tenth object feature point 67, and d(lmks

[64] , limks

[66] ) represents the distance between the eleventh object feature point 64 and the twelfth object feature point 66.

[0077] In a specific embodiment, the analysis feature value also includes a third analysis feature value, which is obtained by analyzing and processing the object feature point information based on a third feature analysis model, wherein the object feature point information also includes a thirteenth object feature point of the second target part, and the third feature analysis model is: S 3 =γ 7 *d 7 +γ 8 *S 4 , S 4 =(γ 9 *S 5 +γ 10 *S 6 ) / γ 11 , S 5 =γ 12 *d 8 +γ 13 *d 9 +γ 14 *d 10 , S 6 =γ 15 *d 11 +γ 16 *d 12 +γ 17 *d 13 , S 3 represents the third analytical eigenvalue, d 7 The distance between the feature points of the third object and the feature points of the third object, d 8 represents the distance between the first object feature point and the seventh object feature point, d 9 represents the distance between the third object feature point and the ninth object feature point, d 10represents the distance between the fifth object feature point and the eleventh object feature point, d 11 represents the distance between the second object feature point and the eighth object feature point, d 12 represents the distance between the fourth object feature point and the tenth object feature point, d 13 represents the distance between the sixth object feature point and the twelfth object feature point, γ 7 , γ 8 , γ 9 , γ 10 , γ 11 , γ 12 , γ 13 , γ 14 , γ 15 , γ 16 and γ 17 Indicates preset parameters.

[0078] Optionally, γ 7 , γ 8 , γ 9 , γ 10 , γ 11 ,γ1 2 , γ 13 , γ 14 , γ 15 , γ 16 and γ 17 It can be set according to actual needs. In a specific embodiment, 0<γ 7 ≤1,1<γ 8 ≤6,0<γ 9 ≤1,0<γ 10 ≤1,4<γ 11 ≤8,0<γ 12 ≤1,0<γ 13 ≤1,0<γ 14 ≤1,0<γ 15 ≤1,0<γ 16 ≤1,0<γ 17 ≤1. Preferably, γ 7 =1,γ 8 =3,γ 9 =1,γ 10 =1,γ 11 =6,γ 12 =1,γ 13 =1,γ 14 =1,γ 15 =1,γ 16 =1,γ 17 =1, then S 3 =d 7 +3*S 4 , S 4 =(S 5+S 6 ) / 6,S 5 =d 8 +d 9 +d 10 , S 6 =d 11 +d 12 +d 13 .

[0079] For example, continue to refer to Figure 6 As shown, the object feature point information also includes the thirteenth object feature point 31 of the nose, and the third feature analysis model is: anchor = d (lmks

[31] , lmks

[52] ) + 3 * S 4 , S 4 =(S 5 +S 6 ) / 6,S 5 =d(lmks

[51] , lmks

[62] )+d(lmks

[52] , lmks

[63] )+d(lmks

[53] , lmks

[64] ), S 6 =d(lmks

[59] , lmks

[68] )+d(lmks

[58] , lmks

[67] )+d(lmks

[57] , lmks

[66] ), where anchor represents the third analysis eigenvalue, d(lmks

[51] , lmks

[62] ) represents the distance between the thirteenth object feature point 31 and the third object feature point 52, d(lmks

[51] , lmks

[62] ) represents the distance between the first object feature point 51 and the seventh object feature point 62, and d(lmks

[52] , limks

[63] ) represents the distance between the third object feature point 52 The distance between the ninth object feature point 63, d(lmks

[53] , lmks

[64] ) represents the distance between the fifth object feature point 53 and the eleventh object feature point 64, d(limks

[59] , lmks

[68] ) represents the distance between the second object feature point 59 and the eighth object feature point 68, d(lmks

[58] , lmks

[67] ) represents the distance between the fourth object feature point 58 and the tenth object feature point 67, and d(lmks

[57] , lmks

[66] ) represents the distance between the sixth object feature point 57 and the twelfth object feature point 66.

[0080] In a specific embodiment, the analysis and processing of the analysis feature value in the analysis feature value information is implemented based on the fourth feature analysis model, and the fourth feature analysis model is: S = (γ 18 *S 1 +Y 19 *S 2 ) / (γ 20 *S 3), S represents the first object feature information, γ 18 , Y 19 and Y 20 Indicates preset parameters.

[0081] Optionally, γ 18 , Y 19 and Y 20 It can be set according to actual needs. In a specific embodiment, 1<γ 18 ≤6,0<γ 5 ≤1,0<γ 6 ≤1, preferably, γ 18 =3,γ 19 =1,γ 20 =1, then: S = (3*S 1 +S 2 ) / S 3 .

[0082] S502: Determine second object feature information based on the object recognition information.

[0083] In a specific embodiment, the step of determining the second object feature information based on the object recognition information specifically includes: determining a dynamic queue of the target object based on the object recognition information; determining the second object feature information based on the dynamic queue of the target object. The dynamic queue is a first-in-first-out (FIFO) queue of a specified length, and the second object feature information is stored in the dynamic queue. The second object feature information stored in the dynamic queue is determined based on historical video frame information.

[0084] Furthermore, in order to improve the accuracy of source object detection, the historical video frame information may be the historical video frame information that is closest to the first video frame information. After determining the second object feature information based on the object recognition information, it also includes: updating the second object feature information stored in the dynamic queue of the target object based on the first object feature information to obtain an updated dynamic queue.

[0085] It should be noted that since the dynamic queue is a first-in-first-out queue of a specified length, after the second object characteristic information stored in the dynamic queue of the target object is updated based on the first object characteristic information, the second object characteristic information first saved in the dynamic queue will be removed from the dynamic queue, so that the second object characteristic information stored in the dynamic queue is always determined based on the historical video frame information that is closest to the first video frame information.

[0086] S503: Determine target recognition result information based on the first object feature information and the second object feature information.

[0087] In a specific embodiment, the step of determining the target recognition result information based on the first object feature information and the second object feature information specifically includes: determining the feature fluctuation information based on the first object feature information and the second object feature information; if the feature fluctuation information is greater than a first threshold, determining that the target recognition result information is the first recognition result information; if the feature fluctuation information is less than or equal to the first threshold, determining that the target recognition result information is the second recognition result information.

[0088] The first recognition result information indicates that the target object is currently speaking, and the second recognition result information indicates that the target object is not speaking at the current moment. The first threshold is a pre-set threshold for measuring whether the target object occurs, and the first threshold can be set according to actual needs. For example, if the first threshold is set to 0.03, if the characteristic fluctuation information is 0.1, the sound detection result of the target object is determined to be the first detection result; if the characteristic fluctuation information is 0.02, the sound detection result of the target object is determined to be the second detection result.

[0089] In a specific embodiment, the feature fluctuation information represents the fluctuation trend of the object feature information of the target object over time. For example, the object feature information represents the aspect ratio of the mouth of the target object. The feature fluctuation information represents the fluctuation trend of the aspect ratio of the mouth of the target object over time. Based on the first object feature information and the second object feature information, the step of determining the feature fluctuation information specifically includes: averaging the first object feature information and the second object feature information to obtain the third object feature information; and determining the feature fluctuation information based on the first object feature information, the second object feature information and the third object feature information. The process of determining the feature fluctuation information can be expressed as: R represents the characteristic fluctuation information, μ represents the characteristic information of the third object, asp i Indicates the first object feature information, asp i-seq Represents the feature information of the second object of the i-seq.

[0090] In a specific implementation, referring to Figure 7 As shown, after determining the target recognition result information based on the object recognition information and the video information in the above step S203, the following steps S601 to S603 may be included, which are specifically as follows:

[0091] S601: Determine whether the target object of the sound is switched based on the target recognition result information.

[0092] In a specific embodiment, based on the target recognition result information, the step of determining whether the target object of the sound has switched specifically includes: based on the target recognition result information, determining the first sound object information; if the first sound object information matches the second sound object information, determining that the target object of the sound has not switched; the second sound object information is the object information of the target object determined to be the sound at the previous moment; if the first sound object information does not match the second sound object information, determining that the target object of the sound has switched.

[0093] Among them, the first sound object information is the object information of the target object determined to be emitting sound at the current moment, and the second sound object information is the object information of the target object determined to be emitting sound at the previous moment. For example, the first sound object information is the ID number of the target object determined to be emitting sound at the current moment, and the second sound object information is the ID number of the target object determined to be emitting sound at the previous moment. If the first sound object information is ID number 001 and the second sound object information is ID number 002, it is determined that the target object of the sound has switched; if the first sound object information is ID number 001 and the second sound object information is ID number 001, it is determined that the target object of the sound has not switched.

[0094] S602: If the target object of the utterance is switched, weight information is determined based on the number of utterances.

[0095] Optionally, the number of utterances is the number of times the target object determined to be uttering at the current moment is determined to be uttering within the observation time window. The observation time window can be set according to actual needs. For example, the target object determined to be uttering at the current moment is user A, and the observation time window includes 10 video frames. In the 10 video frames within the observation time window, user A is determined to be uttering 2 times, then the number of utterances is 2.

[0096] In this embodiment, if the target object of the utterance is switched, the weight information is determined based on the number of utterances. The weight information determination process can be expressed as: α=min(exp(2-n), 1), α represents the weight information, and n represents the number of utterances.

[0097] S603: Determine the sound position information based on the first coordinate information, the second coordinate information and the weight information.

[0098] Optionally, the first coordinate information is the position information of the target object determined to be emitting sound at the current moment, and the second coordinate information is the position information of the target object determined to be emitting sound at the previous moment. For example, the target object determined to be emitting sound at the current moment is user A, and the target object determined to be emitting sound at the previous moment is user B. Then, the first position information is the position information of the center point of the face frame of user A at the current moment, and the second position information is the position information of the center point of the face frame of user B at the previous moment. This embodiment determines the sound position information based on the first coordinate information, the second coordinate information, and the weight information, so that the sound effect can be smoothly transitioned to the target object currently emitting sound, avoiding the disordered drift of the audiovisual sound effect due to some occasional false detections.

[0099] The process of determining the sound location information can be expressed as: SoundLoc(t) * =α·SoundLoc(t-1)+(1-α)SoundLoc(t), SoundLoc(t) * represents the sound position information at time t, α represents the weight information, SoundLoc(t) represents the first coordinate information, and SoundLoc(t-1) represents the second coordinate information.

[0100] After research, the inventor found that when the video processing method provided in this embodiment is applied to the MediaTek MTK-9653SoC, the target object detection in a single frame takes about 5ms, the feature point detection takes about 6ms, and the sound position determination takes about 3ms. The entire process takes no more than 20ms. That is, the video processing method provided in this embodiment can perform real-time sound source object detection, thereby providing real-time surround sound services.

[0101] In summary, the video processing method provided by this implementation scheme obtains video information, determines object recognition information based on the video information, and determines target recognition result information based on the object recognition information and the video information. In this scheme, by determining object recognition information based on video information, and then determining target recognition result information based on object recognition information and video information, real-time sound source object detection can be performed, thereby providing real-time surround sound services. Furthermore, based on the first object feature information and the second object feature information, feature fluctuation information is determined, and then target recognition result information is determined based on the feature fluctuation information, which can improve the accuracy of sound source object detection. If the target object of the sound is switched, weight information is determined based on the number of times the sound is made, and the sound position information is determined based on the first coordinate information, the second coordinate information and the weight information, so that the sound effect can be smoothly transitioned to the target object that is currently making the sound, avoiding the disordered drift of the audiovisual sound effect due to some occasional false detections.

[0102] In order to better implement the video processing method in the embodiment of the present application, based on the video processing method, the embodiment of the present application also provides a video processing device, such as Figure 8 As shown, the video processing device 700 includes:

[0103] An information acquisition module 710 is used to acquire video information;

[0104] An object recognition module 720, configured to determine object recognition information based on the video information;

[0105] The video processing module 730 is used to determine target recognition result information based on the object recognition information and the video information.

[0106] In an embodiment of the present application, by determining object recognition information based on video information, and then determining target recognition result information based on the object recognition information and video information, real-time sound source object detection can be performed, thereby providing real-time surround sound services.

[0107] In some embodiments of the present application, the information acquisition module 710 acquires video information, including:

[0108] Obtaining first video frame information;

[0109] Performing target detection on the first video frame information to obtain first target object information;

[0110] Feature point detection is performed on the first video frame information based on the first target object information to obtain object feature point information.

[0111] In some embodiments of the present application, the video information includes first target object information, and the object recognition module 720 determines the object recognition information based on the video information, including:

[0112] Obtaining the video frame type of the first video frame information;

[0113] Object recognition information is determined based on the video frame type and the first target object information.

[0114] In some embodiments of the present application, the video frame type includes a first type, and the object recognition module 720 determines object recognition information based on the video frame type and the first target object information, including:

[0115] If the video frame type is the first type, determining object identification information based on the first target object information; and / or,

[0116] The video frame type is the second type, and the object recognition module 720 determines the object recognition information based on the video frame type and the first target object information, including:

[0117] Based on the first target object information and the historical video information, object recognition information is determined.

[0118] In some embodiments of the present application, the historical video information includes a plurality of historical video frame information, and the object recognition module 720 determines the object recognition information based on the first target object information and the historical video information, including:

[0119] Determining similarity information based on the first target object information and the historical video frame information in the historical video information;

[0120] Based on the similarity information, object identification information is determined.

[0121] In some embodiments of the present application, the object recognition module 720 determines similarity information based on the first target object information and the historical video frame information in the historical video information, including:

[0122] Performing a first fusion process on the first target object information and the second target object information of the historical video frame information to obtain third target object information;

[0123] Performing a second fusion process on the first target object information and the second target object information to obtain fourth target object information;

[0124] The third target object information and the fourth target object information are divided to obtain similarity information.

[0125] In some embodiments of the present application, the similarity information includes several similarities, and the object recognition module 720 determines the object recognition information based on the similarity information, including:

[0126] Based on a preset similarity threshold, a target similarity is determined from a plurality of similarities, and object recognition information is determined based on the target similarity; the target similarity represents a similarity greater than the similarity threshold; or,

[0127] Based on a preset similarity threshold, if the target similarity cannot be determined from a plurality of similarities, the object recognition information is determined based on the object information of the historical video frame information.

[0128] In some embodiments of the present application, the video information includes object feature point information, and the video processing module 730 determines target recognition result information based on the object recognition information and the video information, including:

[0129] Determining first object feature information based on the object feature point information;

[0130] Determining second object feature information based on the object recognition information;

[0131] Based on the first object feature information and the second object feature information, target recognition result information is determined.

[0132] In some embodiments of the present application, the video processing module 730 determines the first object feature information based on the object feature point information, including:

[0133] Analyze and process the object feature point information to obtain analysis feature value information; the analysis feature value information includes M analysis feature values; M is a positive integer not less than 2;

[0134] The analysis feature values ​​in the analysis feature value information are analyzed and processed to obtain first object feature information.

[0135] In some embodiments of the present application, the analysis characteristic value includes a first analysis characteristic value, a second analysis characteristic value, and a third analysis characteristic value;

[0136] The first analysis characteristic value represents the height of the first target part;

[0137] Accordingly, the first analysis feature value is obtained by the video processing module 730 analyzing and processing the object feature point information based on the first feature analysis model;

[0138] The object feature point information includes the first object feature point, the second object feature point, the third object feature point, the fourth object feature point, the fifth object feature point and the sixth object feature point of the first target part, and the first feature analysis model is: S 1 =γ 1 *d 1 +γ 2 *d 2 +γ 3 *d 3 , S 1 represents the first analytical eigenvalue, d 1 represents the distance between the first object feature point and the second object feature point, d 2 represents the distance between the feature point of the third object and the feature point of the fourth object, d 3 represents the distance between the fifth object feature point and the sixth object feature point, γ 1 , γ 2 and γ 3 Indicates preset parameters;

[0139] The second analysis characteristic value represents the degree of opening and closing of the first target part;

[0140] The second analysis feature value is obtained by the video processing module 730 through analysis and processing of the object feature point information based on the second feature analysis model;

[0141] The object feature point information also includes the seventh object feature point, the eighth object feature point, the ninth object feature point, the tenth object feature point, the eleventh object feature point and the twelfth object feature point of the first target part, and the second feature analysis model is: S 2 =γ 4 *d 4 +γ 5 *d 5 +γ 6 *d 6 , S 2 represents the second analytical eigenvalue, d 4 represents the distance between the seventh object feature point and the eighth object feature point, d 5 represents the distance between the ninth object feature point and the tenth object feature point, d 6 represents the distance between the feature point of the eleventh object and the feature point of the twelfth object, γ 4 , γ 5 and γ 6 Indicates preset parameters;

[0142] The third analysis feature value is obtained by the video processing module 730 through analysis and processing of the object feature point information based on the third feature analysis model;

[0143] The object feature point information also includes a thirteenth object feature point of the second target part, and the third feature analysis model is: S 3 =γ 7 *d 7 +γ 8 *S 4 , S 4 =(γ 9 *S 5 +γ 10 *S 6 ) / γ 11 , S 5 =γ 12 *d 8 +γ 13 *d 9 +γ 14 *d 10 , S 6 =γ 15 *d 11 +γ 16 *d 12 +γ 17 *d 13 , S 3 represents the third analytical eigenvalue, d 7 represents the distance between the third object feature point and the thirteenth object feature point, d 8represents the distance between the first object feature point and the seventh object feature point, dq represents the distance between the third object feature point and the ninth object feature point, d 10 represents the distance between the fifth object feature point and the eleventh object feature point, d 11 represents the distance between the second object feature point and the eighth object feature point, d 12 represents the distance between the fourth object feature point and the tenth object feature point, d 13 represents the distance between the sixth object feature point and the twelfth object feature point, γ 7 , γ 8 , γ 9 , γ 10 , γ 11 , γ 12 , γ 13 , γ 14 , γ 15 , γ 16 and γ 17 Indicates preset parameters;

[0144] Accordingly, the video processing module 730 performs analysis and processing on the analysis feature value in the analysis feature value information based on the fourth feature analysis model;

[0145] Among them, the fourth characteristic analysis model is: S = (γ 18 *S 1 +γ 19 *S 2 ) / (γ 20 *S 3 ), S represents the first object feature information, γ 18 , γ 19 and γ 20 Indicates preset parameters.

[0146] In some embodiments of the present application, the video processing module 730 determines target recognition result information based on the first object feature information and the second object feature information, including:

[0147] Determining characteristic fluctuation information based on the first object characteristic information and the second object characteristic information;

[0148] If the characteristic fluctuation information is greater than the first threshold, the target recognition result information is determined to be the first recognition result information; the first recognition result information indicates that the target object is currently speaking; or,

[0149] If the characteristic fluctuation information is less than or equal to the first threshold, it is determined that the target recognition result information is the second recognition result information; the second recognition result information indicates that the target object has not made any sound at the current moment.

[0150] In some embodiments of the present application, after the video processing module 730 determines the target recognition result information based on the object recognition information and the video information, the video processing module 730 is further configured to:

[0151] Based on the target recognition result information, determining whether the target object of the sound is switched;

[0152] If the target object of the utterance switches, the weight information is determined based on the number of utterances; the number of utterances is the number of times the target object determined to be uttering at the current moment is determined to be uttering within the observation time window;

[0153] The sound position information is determined based on the first coordinate information, the second coordinate information and the weight information; the first coordinate information is the position information of the target object determined to be emitting sound at the current moment, and the second coordinate information is the position information of the target object determined to be emitting sound at the previous moment.

[0154] In some embodiments of the present application, the video processing module 730 determines whether the target object of the sound is switched based on the target recognition result information, including:

[0155] Based on the target recognition result information, first sound object information is determined; the first sound object information is the object information of the target object determined to be the sound at the current moment;

[0156] If the first sound object information matches the second sound object information, it is determined that the target object of the sound has not switched; the second sound object information is the object information of the target object determined to be the sound at the previous moment; or,

[0157] If the first sound object information does not match the second sound object information, it is determined that the target object of the sound is switched.

[0158] The embodiment of the present application further provides a computer device, which integrates any one of the video processing devices provided in the embodiment of the present application, and the computer device includes:

[0159] one or more processors;

[0160] Memory; and

[0161] One or more applications, wherein the one or more applications are stored in the memory and are configured to execute, by the processor, the steps of the video processing method in any of the above-mentioned video processing method embodiments.

[0162] The present application also provides a computer device that integrates any video processing device provided in the present application. Fig. 9 As shown, it shows a schematic diagram of the structure of the computer device involved in the embodiment of the present application, specifically:

[0163] The computer device may include one or more processing core processors 801, one or more computer-readable storage media memories 802, a power supply 803, an input unit 804 and other components. Those skilled in the art will appreciate that Fig. 9 The computer device structure shown in the figure does not constitute a limitation on the computer device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. Among them:

[0164] The processor 801 is the control center of the computer device. It uses various interfaces and lines to connect various parts of the entire computer device. By running or executing software programs and / or modules stored in the memory 802 and calling data stored in the memory 802, it executes various functions of the computer device and processes data, thereby monitoring the computer device as a whole. Optionally, the processor 801 may include one or more processing cores; preferably, the processor 801 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 801.

[0165] The memory 802 can be used to store software programs and modules. The processor 801 executes various functional applications and data processing by running the software programs and modules stored in the memory 802. The memory 802 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 802 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 802 may also include a memory controller to provide the processor 801 with access to the memory 802.

[0166] The computer device also includes a power supply 803 for supplying power to each component. Preferably, the power supply 803 can be logically connected to the processor 801 through a power management system, so that the power management system can manage charging, discharging, power consumption and other functions. The power supply 803 can also include one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, power status indicators and other arbitrary components.

[0167] The computer device may further include an input unit 804, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0168] Although not shown, the computer device may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 801 in the computer device will load the executable files corresponding to the processes of one or more application programs into the memory 802 according to the following instructions, and the processor 801 will run the application programs stored in the memory 802, thereby realizing various functions, as follows:

[0169] Get video information;

[0170] Determining object recognition information based on the video information;

[0171] Based on the object recognition information and the video information, target recognition result information is determined.

[0172] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0173] To this end, the embodiment of the present application provides a computer-readable storage medium, which may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc. A computer program is stored thereon, and the computer program is loaded by a processor to execute the steps in any video processing method provided in the embodiment of the present application. For example, the computer program loaded by the processor can execute the following steps:

[0174] Get video information;

[0175] Determining object recognition information based on the video information;

[0176] Based on the object recognition information and the video information, target recognition result information is determined.

[0177] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the detailed description of other embodiments above, and will not be repeated here.

[0178] In specific implementation, the above units or structures can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above units or structures can refer to the previous method embodiments, which will not be repeated here.

[0179] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0180] The above is a detailed introduction to a video processing method, device and equipment for sound source object detection provided in an embodiment of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A video processing method, characterized in that: include: Get video information; Determining object recognition information based on the video information; Target recognition result information is determined based on the object recognition information and the video information.

2. The method according to claim 1, characterized in that The video information includes first target object information; The determining of object recognition information based on the video information includes: Obtaining a video frame type of the first video frame information; Object recognition information is determined based on the video frame type and the first target object information.

3. The method according to claim 2, characterized in that The video frame type is a first type; and determining the object identification information based on the video frame type and the first target object information includes: determining the object identification information based on the first target object information; and / or, The video frame type is the second type; and determining the object identification information based on the video frame type and the first target object information includes: determining the object identification information based on the first target object information and historical video information.

4. The method according to claim 3, characterized in that The historical video information includes a plurality of historical video frame information; The determining of the object recognition information based on the first target object information and the historical video information includes: Determining similarity information based on the first target object information and the historical video frame information in the historical video information; Based on the similarity information, object identification information is determined.

5. The method according to claim 4, characterized in that The determining of similarity information based on the first target object information and the historical video frame information in the historical video information includes: Performing a first fusion process on the first target object information and the second target object information of the historical video frame information to obtain third target object information; Performing a second fusion process on the first target object information and the second target object information to obtain fourth target object information; The third target object information and the fourth target object information are divided to obtain similarity information.

6. The method according to claim 4, characterized in that The similarity information includes a plurality of similarities, and the object identification information is determined based on the similarity information, including: Based on a preset similarity threshold, a target similarity is determined from the plurality of similarities, and the object identification information is determined based on the target similarity; the target similarity represents the similarity greater than the similarity threshold; or, Based on a preset similarity threshold, if the target similarity cannot be determined from the plurality of similarities, the object recognition information is determined based on the object information of the historical video frame information.

7. The method according to claim 1, characterized in that The video information includes object feature point information, and the target recognition result information is determined based on the object recognition information and the video information, including: Determining first object feature information based on the object feature point information; Determining second object feature information based on the object recognition information; Target recognition result information is determined based on the first object feature information and the second object feature information.

8. The method according to claim 7, characterized in that The determining of first object feature information based on the object feature point information includes: Analyze and process the object feature point information to obtain analysis feature value information; the analysis feature value information includes M analysis feature values; M is a positive integer not less than 2; The analysis feature value in the analysis feature value information is analyzed and processed to obtain first object feature information.

9. The method according to claim 8, characterized in that The analysis characteristic values ​​include a first analysis characteristic value, a second analysis characteristic value and a third analysis characteristic value; The first analysis characteristic value represents the height of the first target part; Accordingly, the first analysis feature value is obtained by analyzing and processing the feature point information of the object based on the first feature analysis model; Wherein, the object feature point information includes a first object feature point, a second object feature point, a third object feature point, a fourth object feature point, a fifth object feature point and a sixth object feature point of a first target part, and the first feature analysis model is: S1=γ1*d1+γ2*d2+γ3*d3, S1 represents the first analysis feature value, d1 represents the distance between the first object feature point and the second object feature point, d2 represents the distance between the third object feature point and the fourth object feature point, d3 represents the distance between the fifth object feature point and the sixth object feature point, and γ1, γ2 and γ3 represent preset parameters; The second analysis characteristic value represents the degree of opening and closing of the first target part; The second analysis feature value is obtained by analyzing and processing the feature point information of the object based on a second feature analysis model; Wherein, the object feature point information also includes the seventh object feature point, the eighth object feature point, the ninth object feature point, the tenth object feature point, the eleventh object feature point and the twelfth object feature point of the first target part, and the second feature analysis model is: S2=γ4*d4+γ5*d5+γ6*d6, S2 represents the second analysis feature value, d4 represents the distance between the seventh object feature point and the eighth object feature point, d5 represents the distance between the ninth object feature point and the tenth object feature point, d6 represents the distance between the eleventh object feature point and the twelfth object feature point, and γ4, γ5 and γ6 represent preset parameters; The third analysis feature value is obtained by analyzing and processing the feature point information of the object based on a third feature analysis model; Wherein, the object feature point information also includes the thirteenth object feature point of the second target part, and the third feature analysis model is: S3 = γ7*d7+γ8*S4, S4 = (γ9*S5+γ 10 *S6) / γ 11 , S5=γ 12 *d8+γ 13 *d9+γ 14 *d 10 , S6 = γ 15 *d 11 +γ 16 *d 12 +γ 17 *d 13 , S3 represents the third analysis feature value, d7 represents the distance between the third object feature point and the thirteenth object feature point, d8 represents the distance between the first object feature point and the seventh object feature point, d9 represents the distance between the third object feature point and the ninth object feature point, d 10 represents the distance between the fifth object feature point and the eleventh object feature point, d 11 represents the distance between the second object feature point and the eighth object feature point, d 12 represents the distance between the fourth object feature point and the tenth object feature point, d 13 represents the distance between the sixth object feature point and the twelfth object feature point, γ7, γ8, γ9, γ 10 , γ 11 , γ 12 , γ 13 , γ 14 , γ 15 , γ 16 and γ 17 Indicates preset parameters; Accordingly, the analysis and processing of the analysis feature value in the analysis feature value information is implemented based on the fourth feature analysis model; Wherein, the fourth characteristic analysis model is: S = (γ 18 *S1+γ 19 *S2) / (γ 20 *S3), S represents the first object feature information, γ 18 , γ 19 and γ 20 Indicates preset parameters.

10. The method according to claim 7, characterized in that The determining target recognition result information based on the first object feature information and the second object feature information includes: determining feature fluctuation information based on the first object feature information and the second object feature information; If the characteristic fluctuation information is greater than a first threshold, the target recognition result information is determined to be first recognition result information; the first recognition result information indicates that the target object is currently speaking; or, If the characteristic fluctuation information is less than or equal to the first threshold, it is determined that the target recognition result information is the second recognition result information; the second recognition result information indicates that the target object has not made any sound at the current moment.

11. The method according to claim 1, characterized in that: The video information includes first target object information and object feature point information; The obtaining of video information includes: Obtaining first video frame information; Performing target detection on the first video frame information to obtain first target object information; Feature point detection is performed on the first video frame information based on the first target object information to obtain object feature point information.

12. The method according to any one of claims 1 to 11, characterized in that: After determining the target recognition result information based on the object recognition information and the video information, the method further includes: Based on the target recognition result information, determining whether the target object of the sound is switched; If the target object making the sound switches, weight information is determined based on the number of times the sound is made; the number of times the target object determined to be making the sound at the current moment is determined to be making the sound within the observation time window; The sound position information is determined based on the first coordinate information, the second coordinate information and the weight information; the first coordinate information is the position information of the target object determined to be emitting sound at the current moment, and the second coordinate information is the position information of the target object determined to be emitting sound at the previous moment.

13. The method according to claim 12, characterized in that The determining, based on the target recognition result information, whether the target object of the sound is switched includes: Based on the target recognition result information, first sound object information is determined; the first sound object information is the object information of the target object determined to be the sound at the current moment; If the first sound object information matches the second sound object information, it is determined that the target object of the sound has not switched; the second sound object information is the object information of the target object determined to be the sound at the previous moment; or, If the first sound object information does not match the second sound object information, it is determined that the sound target object is switched.

14. A video processing device, characterized in that: include: An information acquisition module, used for acquiring video information; An object recognition module, configured to determine object recognition information based on the video information; A video processing module, used to determine target recognition result information based on the object recognition information and the video information; Preferably, the information acquisition module acquires video information, including: Obtaining first video frame information; Performing target detection on the first video frame information to obtain first target object information; Performing feature point detection on the first video frame information based on the first target object information to obtain object feature point information; Preferably, the video information includes first target object information, and the object recognition module determines the object recognition information based on the video information, including: Obtaining a video frame type of the first video frame information; Determining object recognition information based on the video frame type and the first target object information; Preferably, the video frame type is a first type, and the object recognition module determines object recognition information based on the video frame type and the first target object information, including: If the video frame type is the first type, determining object identification information based on the first target object information; and / or, The video frame type is the second type, and the object recognition module determines the object recognition information based on the video frame type and the first target object information, including: Determining object recognition information based on the first target object information and historical video information; Preferably, the historical video information includes a plurality of historical video frame information, and the object recognition module determines the object recognition information based on the first target object information and the historical video information, including: Determining similarity information based on the first target object information and the historical video frame information in the historical video information; Determining object identification information based on the similarity information; Preferably, the object recognition module determines similarity information based on the first target object information and the historical video frame information in the historical video information, including: Performing a first fusion process on the first target object information and the second target object information of the historical video frame information to obtain third target object information; Performing a second fusion process on the first target object information and the second target object information to obtain fourth target object information; Performing a division process on the third target object information and the fourth target object information to obtain similarity information; Preferably, the similarity information includes a plurality of similarities, and the object recognition module determines the object recognition information based on the similarity information, including: Based on a preset similarity threshold, determine a target similarity from the plurality of similarities, and determine the object recognition information based on the target similarity; the target similarity represents a similarity greater than the similarity threshold; or, Based on a preset similarity threshold, if the target similarity cannot be determined from the plurality of similarities, determining the object identification information based on the object information of the historical video frame information; Preferably, the video information includes object feature point information, and the video processing module determines target recognition result information based on the object recognition information and the video information, including: Determining first object feature information based on the object feature point information; Determining second object feature information based on the object recognition information; Determining target recognition result information based on the first object feature information and the second object feature information; Preferably, the video processing module determines the first object feature information based on the object feature point information, including: Analyze and process the object feature point information to obtain analysis feature value information; the analysis feature value information includes M analysis feature values; M is a positive integer not less than 2; Analyzing and processing the analysis feature value in the analysis feature value information to obtain first object feature information; Preferably, the analysis characteristic value includes a first analysis characteristic value, a second analysis characteristic value and a third analysis characteristic value; The first analysis characteristic value represents the height of the first target part; Accordingly, the first analysis feature value is obtained by the video processing module analyzing and processing the object feature point information based on the first feature analysis model; Wherein, the object feature point information includes a first object feature point, a second object feature point, a third object feature point, a fourth object feature point, a fifth object feature point and a sixth object feature point of a first target part, and the first feature analysis model is: S1=γ1*d1+γ2*d2+γ3*d3, S1 represents the first analysis feature value, d1 represents the distance between the first object feature point and the second object feature point, d2 represents the distance between the third object feature point and the fourth object feature point, d3 represents the distance between the fifth object feature point and the sixth object feature point, and γ1, γ2 and γ3 represent preset parameters; The second analysis characteristic value represents the degree of opening and closing of the first target part; The second analysis feature value is obtained by the video processing module analyzing and processing the feature point information of the object based on the second feature analysis model; Wherein, the object feature point information also includes the seventh object feature point, the eighth object feature point, the ninth object feature point, the tenth object feature point, the eleventh object feature point and the twelfth object feature point of the first target part, and the second feature analysis model is: S2=γ4d4+γ5*d5+γ6*d6, S2 represents the second analysis feature value, d4 represents the distance between the seventh object feature point and the eighth object feature point, d5 represents the distance between the ninth object feature point and the tenth object feature point, d6 represents the distance between the eleventh object feature point and the twelfth object feature point, and γ4, γ5 and γ6 represent preset parameters; The third analysis feature value is obtained by the video processing module analyzing and processing the object feature point information based on the third feature analysis model; Wherein, the object feature point information also includes the thirteenth object feature point of the second target part, and the third feature analysis model is: S3 = γ7*d7+γ8*S4, S4 = (γ9*S5+γ 10 *S6) / γ 11 , S5=γ 12 *d8+γ 13 *d9+γ 14 *d 10 , S6 = γ 15 *d 11 +γ 16 *d 12 +γ 17 *d 13 , S3 represents the third analysis feature value, d7 represents the distance between the third object feature point and the thirteenth object feature point, d8 represents the distance between the first object feature point and the seventh object feature point, d9 represents the distance between the third object feature point and the ninth object feature point, d 10 represents the distance between the fifth object feature point and the eleventh object feature point, d 11 represents the distance between the second object feature point and the eighth object feature point, d 12 represents the distance between the fourth object feature point and the tenth object feature point, d 13 represents the distance between the sixth object feature point and the twelfth object feature point, γ7, γ8, γ9, γ 10 , γ 11 , γ 12 , γ 13 , γ 14 , γ 15 , γ 16 and γ 17 Indicates preset parameters; Accordingly, the video processing module performs analysis and processing on the analysis feature value in the analysis feature value information based on the fourth feature analysis model; Wherein, the fourth characteristic analysis model is: S = (γ 18 *S1+γ 19 *S2) / (γ 20 *S3), S represents the first object feature information, γ 18 , γ 19 and γ 20 Indicates preset parameters; Preferably, the video processing module determines target recognition result information based on the first object feature information and the second object feature information, including: determining feature fluctuation information based on the first object feature information and the second object feature information; If the characteristic fluctuation information is greater than a first threshold, the target recognition result information is determined to be first recognition result information; the first recognition result information indicates that the target object is currently speaking; or, If the characteristic fluctuation information is less than or equal to the first threshold, determining that the target recognition result information is second recognition result information; the second recognition result information indicates that the target object is not making a sound at the current moment; Preferably, after the video processing module determines the target recognition result information based on the object recognition information and the video information, the video processing module is further used to: Based on the target recognition result information, determining whether the target object of the sound is switched; If the target object making the sound switches, weight information is determined based on the number of times the sound is made; the number of times the target object determined to be making the sound at the current moment is determined to be making the sound within the observation time window; Determine the sound position information based on the first coordinate information, the second coordinate information and the weight information; the first coordinate information is the position information of the target object determined to be making a sound at the current moment, and the second coordinate information is the position information of the target object determined to be making a sound at the previous moment; Preferably, the video processing module determines whether the target object of the sound is switched based on the target recognition result information, including: Based on the target recognition result information, first sound object information is determined; the first sound object information is the object information of the target object determined to be the sound at the current moment; If the first sound object information matches the second sound object information, it is determined that the target object of the sound has not switched; the second sound object information is the object information of the target object determined to be the sound at the previous moment; or, If the first sound object information does not match the second sound object information, it is determined that the sound target object is switched.

15. A computer device, characterized in that: The computer device comprises: one or more processors; Memory; and One or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the processor to implement the video processing method according to any one of claims 1 to 13.

16. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and the computer program is loaded by a processor to execute the steps in the video processing method according to any one of claims 1 to 13.

Images