A target tracking method, device, equipment and storage medium

Abstract

The present application provides a target tracking method, device, equipment and storage medium, the target tracking method comprising: performing storyboard processing on the input script video to obtain multiple storyboard segments constituting the script video; performing face detection and body detection on the multiple storyboard segments respectively, and determining the information of the characters appearing in the multiple storyboard segments according to the face detection result and body detection result of each storyboard segment; performing face detection and body detection on each frame image contained in the script video, and determining the target character appearing in the frame image from the character information according to the face detection result and body detection result of the frame image. In this way, the present application can achieve high-quality, fully automatic human body tracking by combining the advantages of multiple character tracking methods such as face detection and body detection, and is applicable to multiple application scenarios, effectively overcoming the defect that the prior art is not suitable for processing script videos involving multi-lens switching.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a target tracking method, device, equipment and storage medium. Background Art

[0002] Human tracking is an important research direction in the field of computer vision and machine learning. Its goal is to detect and track the human body from video sequences or real-time video streams, and to achieve real-time tracking of the human body's position, posture, and behavior.

[0003] At present, the ReID (Person re-identification) method is commonly used in the prior art to track people captured in surveillance videos. However, since the ReID method is prone to recognition failure in scenes where the person's posture changes greatly between shots or where less body parts are exposed (e.g., only the image above the person's shoulders), the ReID method in the prior art is not suitable for tracking people in scripted videos involving multi-shot switching (e.g., film and television drama clips, role-playing videos, etc.). Summary of the invention

[0004] In view of this, the present application provides a target tracking method, device, equipment and storage medium, which can achieve high-quality, fully automatic human body tracking by combining the advantages of various character tracking methods such as face detection and body detection, and is suitable for a variety of application scenarios, effectively overcoming the defect that the existing technology is not suitable for processing script-type videos involving multi-lens switching.

[0005] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings.

[0006] In a first aspect, an embodiment of the present application provides a target tracking method, the target tracking method comprising:

[0007] Performing storyboard processing on the input script video to obtain a plurality of storyboard segments constituting the script video;

[0008] Performing face detection and body detection on the multiple storyboard segments respectively, and determining information of characters appearing in the multiple storyboard segments according to the face detection result and the body detection result of each storyboard segment;

[0009] Face detection and body detection are performed on each frame image contained in the script video, and according to the face detection result and body detection result of the frame image, a target character contained in the frame image is determined from the character information.

[0010] In a second aspect, an embodiment of the present application provides a target tracking device, the target tracking device comprising:

[0011] A storyboard module, used for storyboarding an input script video to obtain a plurality of storyboard segments constituting the script video;

[0012] A detection module, configured to perform face detection and body detection on the plurality of storyboard segments respectively, and determine information of characters appearing in the plurality of storyboard segments according to the face detection result and the body detection result of each storyboard segment;

[0013] The matching module is used to perform face detection and body detection on each frame image contained in the script video, and determine the target character contained in the frame image from the character information according to the face detection result and the body detection result of the frame image.

[0014] In a third aspect, an embodiment of the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned target tracking method when executing the computer program.

[0015] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above-mentioned target tracking method are executed.

[0016] The technical solution provided by the embodiments of the present application may have the following beneficial effects:

[0017] The target tracking method, device, equipment and storage medium provided by the embodiment of the present application perform storyboard processing on the input script video to obtain multiple storyboard segments constituting the script video; perform face detection and body detection on the multiple storyboard segments respectively, and determine the information of the characters appearing in the multiple storyboard segments according to the face detection result and body detection result of each storyboard segment; perform face detection and body detection on each frame image contained in the script video, and determine the target character appearing in the frame image from the character information according to the face detection result and body detection result of the frame image. In this way, the present application can achieve high-quality, fully automatic human body tracking by combining the advantages of multiple character tracking methods such as face detection and body detection, and is suitable for a variety of application scenarios, effectively overcoming the defect that the prior art is not suitable for processing script videos involving multi-lens switching. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 A schematic diagram of a target tracking method is shown;

[0020] Figure 2 A schematic flow chart of a method for determining information of characters appearing in multiple storyboard segments provided by an embodiment of the present application is shown;

[0021] Figure 3 A flow chart of a method for determining a target character included in each frame of an image provided by an embodiment of the present application is shown;

[0022] Figure 4 A schematic diagram of the structure of a target tracking device provided in an embodiment of the present application is shown;

[0023] Figure 5 A schematic diagram of the structure of an electronic device 500 provided in an embodiment of the present application. DETAILED DESCRIPTION

[0024] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of explanation and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in this application shows the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowchart can be implemented out of sequence, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart under the guidance of the content of the present application, or remove one or more operations from the flowchart.

[0025] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0026] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the existence of the features declared thereafter, but does not exclude the addition of other features.

[0027] At present, the ReID method is commonly used in the prior art to track people captured in surveillance videos. However, since the ReID method is prone to recognition failure in scenes where the person's posture changes greatly between shots or where less body parts are exposed, the ReID method in the prior art is not suitable for tracking people in scripted videos involving multiple shot switching.

[0028] Based on this, the present application provides a target tracking method, device, equipment and storage medium. By combining the advantages of various character tracking methods such as face detection and body detection, high-quality, fully automatic human body tracking can be achieved, and it is suitable for a variety of application scenarios, effectively overcoming the defect that the existing technology is not suitable for processing script-type videos involving multi-shot switching.

[0029] In one embodiment of the present application, a target tracking method can be applied to video editing software that can produce script videos, and can also be applied to games that allow users to customize and create videos, and the above-mentioned video editing software and the above-mentioned game can be run on a terminal device or a server; wherein the terminal device can be a local terminal device, and when the above-mentioned video editing software and the above-mentioned game are run on the server, the target tracking method can be implemented and executed based on a cloud interaction system, wherein the cloud interaction system includes a server and a client device (i.e., a terminal device).

[0030] To facilitate understanding of the embodiments of the present application, a target tracking method, device, equipment and storage medium provided in the embodiments of the present application are described in detail below.

[0031] Reference Figure 1 As shown, Figure 1 A schematic flow chart of a target tracking method provided in an embodiment of the present application is shown, wherein the target tracking method includes steps S101-S103; specifically:

[0032] S101, performing storyboard processing on the input script video to obtain a plurality of storyboard segments constituting the script video.

[0033] S102, performing face detection and body detection on the multiple storyboard segments respectively, and determining information of characters appearing in the multiple storyboard segments according to the face detection result and the body detection result of each storyboard segment.

[0034] S103, performing face detection and body detection on each frame image contained in the script video, and determining the target character contained in the frame image from the character information according to the face detection result and the body detection result of the frame image.

[0035] The target tracking method provided in the embodiment of the present application performs storyboard processing on the input script video to obtain multiple storyboard segments that constitute the script video; performs face detection and body detection on the multiple storyboard segments respectively, and determines the information of the characters appearing in the multiple storyboard segments according to the face detection results and body detection results of each storyboard segment; performs face detection and body detection on each frame image contained in the script video, and determines the target character appearing in the frame image from the character information according to the face detection results and body detection results of the frame image. In this way, the present application can achieve high-quality, fully automatic human body tracking by combining the advantages of multiple character tracking methods such as face detection and body detection, and is suitable for a variety of application scenarios, effectively overcoming the defect that the prior art is not suitable for processing script videos involving multi-lens switching.

[0036] The following is an exemplary description of each step in the target tracking method provided in the embodiment of the present application:

[0037] S101, performing storyboard processing on the input script video to obtain a plurality of storyboard segments constituting the script video.

[0038] Here, a graphical user interface of a target application (such as video editing software, games that allow users to customize and create videos, etc.) is provided through a terminal device, and a video upload area is displayed on the graphical user interface. As an optional embodiment, the user can click on the video upload area to trigger a path selection window to be displayed on the graphical user interface, and select the target path corresponding to the expected input script video from the path selection window to complete the input of the script video; as another optional embodiment, the user can also directly drag the expected input script video to the above-mentioned video upload area to complete the input of the script video; the embodiment of the present application does not impose any limitation on the specific input method of the above-mentioned script video.

[0039] Specifically, the above-mentioned script video is a video involving multi-shot switching, that is, the script video includes multiple characters. According to the development of the plot created by the user, each character can be transitioned in different video shooting scenes (that is, the script video involves switching between multiple video shooting scenes), and each character will also have a variety of different shooting shots such as close-ups (such as close-up shots of the character's face, half-body shots, etc.) and long shots (such as full-body shots of the character, back shots, etc.); the present application embodiment does not impose any limitation on the specific video content of the above-mentioned script video.

[0040] It should be noted that the above-mentioned script video can be a video shot by real people in a real scene (for example, the script video can be a film and television drama clip, a short video shot by the user himself, etc.), or it can be a video created by the user in the game based on multiple game characters. The specific creation form of the above-mentioned script video and the specific characters appearing in the script video are not limited in any way in the embodiments of the present application.

[0041] Here, when the input script video is storyboarded, multiple continuous image frames with smaller changes in the lens picture can be divided into a storyboard segment according to the changes in the lens picture, thereby realizing automatic storyboarding of the script video and splitting the script video into multiple storyboard segments.

[0042] Specifically, as an optional embodiment, when executing the above step S101, the input script video may be storyboarded according to the method shown in the following steps a1-a2:

[0043] Step a1: by detecting the pixel difference between adjacent frames in the script video, determine from the script video a target adjacent frame whose pixel difference is greater than or equal to a preset pixel difference threshold.

[0044] Here, when executing step a1, the main algorithm principle of the underlying algorithm is to calculate the pixel difference between adjacent frames in the script video. When the calculated pixel difference is greater than or equal to the preset pixel difference threshold, it is considered that a shot switch has occurred in the script video (which is also equivalent to a shooting scene switch), so as to determine that video segmentation needs to be performed between adjacent frames (i.e., the above-mentioned target adjacent frames) whose pixel difference is greater than or equal to the preset pixel difference threshold, so as to ensure that the shot picture changes between the multiple continuous image frames in each storyboard segment obtained after segmentation are small (i.e., less than the above-mentioned preset pixel difference threshold).

[0045] It should be noted that the specific value of the above-mentioned preset pixel difference threshold can be set according to the actual storyboard requirements, and the embodiments of the present application do not impose any limitation on this.

[0046] Step a2: using each of the target adjacent frames as a video segmentation point, segmenting the script video to obtain the multiple storyboard segments.

[0047] Specifically, after determining the target adjacent frames whose pixel differences are greater than or equal to the above-mentioned preset pixel difference threshold, for each target adjacent frame, video segmentation can be performed between the target adjacent frames (that is, the video segmentation point is performed between the target adjacent frames), and the previous image frame in the target adjacent frame is segmented into the previous storyboard segment, and the next image frame in the target adjacent frame is segmented into the next storyboard segment, so as to obtain multiple storyboard segments of the script video [(st1, ed1), (st2, ed2), ... (stn, edn)], wherein sti represents the starting frame of the i-th storyboard segment, edi represents the ending frame of the i-th storyboard segment, and the maximum value of i is n (that is, the number of segments of the above storyboard segment is n), and the specific value of n depends on the actual storyboard processing result of the script video, which is not limited in any way in the embodiment of the present application.

[0048] It should be noted that the specific execution tool of the underlying algorithm mentioned in the above step a1 is not unique. For example, the existing mature tool scenedetect (a command line tool for video scene detection that can analyze video content, automatically detect scene changes in the video, and divide the video into multiple segments) can be used to execute the above steps a1-a2, or the above steps a1-a2 can be executed by other tools. The embodiments of the present application do not impose any limitations on this.

[0049] S102, performing face detection and body detection on the multiple storyboard segments respectively, and determining information of characters appearing in the multiple storyboard segments according to the face detection result and the body detection result of each storyboard segment.

[0050] Here, when executing step S102, face detection and body detection are performed on each storyboard segment respectively to obtain face detection results and body detection results for each storyboard segment, and then the detected face detection results and body detection results are matched to obtain multiple groups of matched face detection results and body detection results. Each group of matched face detection results and body detection results corresponds to a complete character. According to the face detection result corresponding to each character, the face identity features of each character can be determined. According to the body detection result corresponding to each character, the body identity features of each character can be determined. Therefore, the face identity features and body identity features of each character can be used as the character identity information corresponding to each character. The characters and their character identity information appearing in all storyboard segments (i.e., the multiple storyboard segments) are summarized to obtain the character information in the multiple storyboard segments.

[0051] Specifically, when executing step S102, as an optional embodiment, face detection and body detection may be performed on all image frames included in each storyboard segment.

[0052] Specifically, when executing step S102, considering that the pixel differences between different image frames in the same storyboard segment are small (that is, the changes in the lens picture are small), therefore, as another optional embodiment, for each storyboard segment, a preset number of image frames (for example, 3 image frames) can be extracted from the storyboard segment as representative image frames of the storyboard segment, and face detection and body detection are performed on each representative image frame, so as to use the face detection result and body detection result of each representative image frame as the face detection result and body detection result of the storyboard segment, so as to reduce the image processing burden and improve the image processing efficiency; wherein, the embodiment of the present application does not impose any limitation on the specific value of the above preset number.

[0053] It should be noted that when extracting a preset number of representative image frames to represent storyboard clips, since the representative image frames of each storyboard clip basically summarize the video content of the entire script video, that is, even if only face detection and body detection are performed on the extracted representative image frames, the obtained face detection results and body detection results can cover all the characters appearing in the script video.

[0054] S103, performing face detection and body detection on each frame image contained in the script video, and determining the target character contained in the frame image from the character information according to the face detection result and the body detection result of the frame image.

[0055] Here, the above-mentioned character information determined in the above-mentioned step S102 is equivalent to covering all the characters in the script video and the character identity information of each character (that is, the facial identity features and body identity features of each character). Therefore, when executing step S103, the script video is detected frame by frame (that is, face detection and body detection are performed on each frame of the script video). After the face detection result and body detection result of each frame of the image are obtained, the character identity information of the candidate characters appearing in each frame of the image (that is, the characters whose attribution has not yet been determined) can be determined in a processing method similar to the above-mentioned step S102. Therefore, through information matching, the characters that match the character identity information of the above-mentioned candidate characters can be determined from the above-mentioned character information as the target characters contained in each frame of the image (equivalent to determining the specific characters corresponding to the above-mentioned candidate characters among all the above-mentioned characters).

[0056] Specifically, similar to the above-mentioned step S102, for each frame image contained in the script video, after performing face detection and body detection on the frame image, the detected face detection result can be matched with the body detection result to obtain multiple groups of matched face detection results and body detection results, each group of matched face detection results and body detection results corresponds to a complete candidate character, and based on the face detection result corresponding to each candidate character, the face identity features of each candidate character can be determined, and based on the body detection result corresponding to each candidate character, the body identity features of each candidate character can be determined, so that the face identity features and body identity features of each candidate character can be used as the character identity information corresponding to each candidate character.

[0057] The specific implementation process of each of the above steps in the embodiment of the present application is described in detail below:

[0058] Regarding the specific implementation process of the above step S102, in an optional implementation mode, Figure 2 A flow chart of a method for determining information of characters appearing in multiple storyboard segments provided by an embodiment of the present application is shown. Figure 2 As shown, when executing the above step S102, the method includes steps S201-S203, specifically:

[0059] S201, for each storyboard segment, determine the characters appearing in the storyboard segment according to the face detection result and the body detection result of the storyboard segment.

[0060] Here, referring to the relevant instructions at the above-mentioned step S102, it can be known that when executing step S201, face detection and body detection can be performed on all image frames contained in each storyboard segment; or a preset number of image frames (for example, 3 image frames) can be extracted from each storyboard segment as representative image frames of each storyboard segment, and face detection and body detection can be performed on the representative image frames of each storyboard segment.

[0061] Specifically, for each storyboard segment, when performing face detection on the storyboard segment, multiple images to be detected of the storyboard segment (that is, each image frame or each representative image frame contained in the storyboard segment) can be input into a face detection algorithm (that is, a face detection model), and the face image contained in each image to be detected is detected by the face detection algorithm, and the image to be detected with a face position mark (such as a face position detection box) is output as the face detection result of the storyboard segment.

[0062] It should be noted that the above-mentioned face detection algorithm is not unique. For example, the RetinaFace face detection algorithm can be selected, or other face detection algorithms such as the MTCNN (Multi-task Cascaded Convolutional Networks) face detection algorithm can be selected. The embodiments of the present application do not impose any limitation on the specific algorithm content of the above-mentioned face detection algorithm.

[0063] Specifically, for each storyboard segment, when performing human body detection on the storyboard segment, multiple images to be detected of the storyboard segment (that is, each image frame or each representative image frame contained in the storyboard segment) can be input into a human body detection algorithm (that is, a human body detection model), and the human body image contained in each image to be detected is detected by the human body detection algorithm, and the image to be detected with a human body position mark (such as a human body position detection box) is output as the human body detection result of the storyboard segment.

[0064] It should be noted that the above-mentioned human body detection algorithm is not unique. For example, you can choose the yolov9 detection algorithm (that is, one of the latest yolo series algorithms), or you can choose other versions of the yolo series detection algorithm. The yolo series detection algorithm (including the yolov9 detection algorithm) is used to identify the object category and object position contained in the input image to be detected (the image area where the object is located in the image to be detected is circled by the position detection frame). By screening the detected object category as the human body category, the human body position detection frame (that is, the human body position mark) carried in the image to be detected can be retained; the embodiment of the present application does not make any limitation on the specific algorithm content of the above-mentioned human body detection algorithm.

[0065] Here, referring to the relevant description of step S102, it can be known that in step S201, each of the characters corresponds to a set of matching face detection results and body detection results; wherein, for each storyboard segment, according to the face position detection frame (i.e., the face detection result of the storyboard segment) and the body position detection frame (i.e., the body detection result of the storyboard segment) carried by each of the images to be detected in the storyboard segment, the face position detection frame and the body position detection frame in the storyboard segment can be matched according to the following three optional implementations, specifically:

[0066] Optional implementation method 1: Taking a storyboard clip as an example, for multiple human position detection frames (i.e., human detection results) contained in the storyboard clip, the number of face position detection frames (i.e., face detection results) contained in each human position detection frame can be counted; if a human position detection frame only contains one face position detection frame, it can be determined that the face position detection frame is a face position detection frame that matches the human position detection frame.

[0067] Optional implementation 2: If a human body position detection frame does not contain any face position detection frame (equivalent to a human body position detection frame containing zero face position detection frames), it can be determined that the human body position detection frame has no matching face position detection frame.

[0068] Optional implementation 3: If a human body position detection frame contains multiple face position detection frames, then based on the proximity principle, a face position detection frame that is closest to the human body position detection frame can be determined as the face position detection frame that matches the human body position detection frame.

[0069] It should be noted that when there is an intersection between a face position detection frame and a body position detection frame, it can be determined that the face position detection frame is included in the body position detection frame; or when the intersection between a face position detection frame and a body position detection frame is greater than or equal to a preset image area threshold, it can be determined that the face position detection frame is included in the body position detection frame. The specific method for determining whether a body position detection frame includes a face position detection frame is not limited in any way in the embodiments of the present application.

[0070] S202, determining the facial identity features and the human body identity features corresponding to the character according to the face detection results and the human body detection results corresponding to the character, and using the facial identity features and the human body identity features as the character identity information of the character.

[0071] Here, referring to the specific implementation method of the above-mentioned step S201, it can be known that each character corresponds to a set of matching face detection results and body detection results, wherein the face detection result can be specifically represented as a face position detection frame (i.e., face position mark) carried in the above-mentioned image to be detected (i.e., each image frame contained in a storyboard fragment or each representative image frame), and the body detection result can be specifically represented as a body position detection frame (i.e., body position mark) carried in the above-mentioned image to be detected.

[0072] Specifically, for each character appearing, the face position detection frame corresponding to the character (i.e., the face detection result) can be cut out from the above-mentioned image to be detected, and the face position detection frame corresponding to the character can be input into the face recognition model, and the face recognition model can output the face identity feature vector corresponding to the face position detection frame as the face identity feature corresponding to the character.

[0073] For example, taking the Arcface model as the face recognition model, the face position detection frame corresponding to the character is input into the Arcface model. Since the Arcface model has been trained in an open source large-scale face recognition dataset, the output result of the penultimate neural network of the Arcface model can be directly obtained as the face identity feature vector, and the face identity feature vector obtained is used as the face identity feature corresponding to the character.

[0074] Specifically, for each character appearing, the human body position detection frame (i.e., human body detection result) corresponding to the character can be cut out from the above-mentioned image to be detected, and the human body position detection frame corresponding to the character appearing can be input into the human body recognition model, and the human body recognition model can output the human body identity feature vector corresponding to the human body position detection frame as the human body identity feature corresponding to the character appearing.

[0075] For example, taking the above-mentioned human recognition model as a ReID model, the human position detection frame corresponding to the person appearing in the scene is input into the ReID model, and the output result of the penultimate neural network layer of the ReID model can be directly obtained as the above-mentioned human identity feature vector (i.e., the ReID feature vector), and the obtained above-mentioned human identity feature vector is used as the human identity feature corresponding to the person appearing in the scene.

[0076] S203, performing deduplication processing on the character identity information of the characters appearing in the multiple storyboard segments, and obtaining the deduplication-processed character identity information of the multiple characters appearing as the appearing character information.

[0077] Here, for all the characters appearing in the above-mentioned multiple storyboard segments, the character identity information of the first character to appear can be directly added to the above-mentioned character information as the character identity information that is determined to be non-repetitive.

[0078] Here, for other characters that appear after the first character in the above-mentioned multiple storyboard segments, it can be determined whether the character identity information of the other characters is repeated with the character identity information of the characters already added in the above-mentioned character information. If it is determined that there is a duplication, the character identity information of the above-mentioned other characters to be added will be discarded; if it is determined that there is no duplication, the character identity information of the above-mentioned other characters to be added can be added to the above-mentioned character information.

[0079] Specifically, when judging whether the character identity information of the other characters is repeated with the character identity information of the characters already added in the above-mentioned character information, the feature distances (such as Euclidean distance, cosine distance, etc.) between the facial identity features of the other characters and the facial identity features of the added characters and the feature distances (such as Euclidean distance, cosine distance, etc.) between the body identity features of the other characters and the body identity features of the added characters can be calculated respectively, and from the above-mentioned calculation results, it is determined whether the minimum value of the feature distance exceeds the preset feature distance threshold; if it exceeds, it can be determined that the character identity information of the other characters is not repeated with the character identity information of the added characters; if it does not exceed, it can be determined that the character identity information of the other characters is repeated with the character identity information of the added characters.

[0080] It should be noted that when executing step S203, in addition to calculating the characteristic distances between the facial identity features of the other characters and the facial identity features of the added characters, as well as the characteristic distances between the body identity features of the other characters and the body identity features of the added characters, as in the above-mentioned specific implementation method, as another optional embodiment, in order to save computing resources, when the face image quality is high (that is, the confidence of the face identity features is higher than a preset threshold), the characteristic distances between the facial identity features of the other characters and the facial identity features of the added characters can be calculated; when the face image quality is low (that is, the confidence of the face identity features is lower than a preset threshold), the characteristic distances between the body identity features of the other characters and the body identity features of the added characters can be calculated; the embodiment of the present application does not impose any limitation on the specific deduplication method of the deduplication processing in the above-mentioned step S203.

[0081] Regarding the specific implementation process of the above step S103, in an optional implementation mode, Figure 3 A flow chart of a method for determining a target character included in each frame of an image provided by an embodiment of the present application is shown. Figure 3 As shown, when executing the above step S103, the method includes steps S301-S303, specifically:

[0082] S301, determining candidate characters in the frame image according to the face detection result and the body detection result of the frame image.

[0083] Here, the specific implementation of step S301 is similar to the specific implementation of the aforementioned step S201, wherein, similar to the characters appearing in step S201, in step S301, each of the candidate characters also corresponds to a set of matching face detection results and body detection results; the specific implementation of step S301 can refer to the specific implementation of the aforementioned step S201, and the repeated parts will not be repeated here.

[0084] S302, determining candidate facial identity features and candidate human body identity features corresponding to the candidate character according to the face detection results and human body detection results corresponding to the candidate character, and using the candidate facial identity features and the candidate human body identity features as character identity information of the candidate character.

[0085] Here, the specific implementation of step S302 is similar to the specific implementation of the aforementioned step S202, wherein the specific implementation of step S302 can refer to the specific implementation of the aforementioned step S202, and the repeated parts are not repeated here.

[0086] S303: Determine, from the information of the characters, a character whose identity information matches the candidate characters as the target character.

[0087] Here, when executing step S303, as an optional embodiment, the target character matching each candidate character may be determined according to the method shown in steps b1-b3 below, specifically:

[0088] Step b1: judging whether the confidence of the face detection result meets a preset confidence judgment condition according to the face detection result corresponding to the candidate character.

[0089] Here, when executing step b1, as an optional embodiment, it is possible to determine whether the confidence of the face detection result corresponding to each candidate character meets the preset confidence determination condition according to the method shown in the following steps c1-c3, specifically:

[0090] Step c1: judging, based on the face detection result corresponding to the candidate character, whether the image proportion of the face image of the candidate character in the frame image is greater than or equal to a preset image proportion.

[0091] Here, the face detection result corresponding to the candidate character can be a face position detection box in the image frame. By calculating the ratio between the area of ​​the face position detection box and the overall image area of ​​the image frame, the image proportion of the face image of the candidate character in the frame image can be obtained.

[0092] It should be noted that the specific value of the above-mentioned preset image proportion can be set according to the actual confidence screening requirements, and the embodiments of the present application do not impose any limitation on this.

[0093] Step c2: Under the condition that it is determined that the image proportion of the facial image of the candidate character in the frame image is greater than or equal to the preset image proportion, determine whether the image clarity of the facial image of the candidate character is greater than or equal to the preset image clarity.

[0094] Here, if the image proportion of the facial image of the candidate character in the frame image is less than the above-mentioned preset image proportion, it can be directly determined that the confidence of the face detection result does not meet the above-mentioned preset confidence judgment condition, and jump to execute the subsequent step b3.

[0095] Here, if the image proportion of the facial image of the candidate character in the frame image is greater than or equal to the above-mentioned preset image proportion, it is necessary to continue to determine whether the image clarity of the facial image of the candidate character is greater than or equal to the preset image clarity.

[0096] It should be noted that the specific value of the above-mentioned preset image clarity can be set according to the actual confidence screening requirements, and the embodiments of the present application do not impose any limitation on this.

[0097] Step c3: If it is determined that the image definition of the facial image of the candidate character is greater than or equal to the preset image definition, determine whether the confidence of the face detection result corresponding to the candidate character meets the preset confidence judgment condition.

[0098] Here, if the image proportion of the facial image of the candidate character in the frame image and the above-mentioned image clarity both meet the preset conditions (that is, the image proportion of the facial image of the candidate character in the frame image is greater than or equal to the above-mentioned preset image proportion and the image clarity of the facial image of the candidate character is greater than or equal to the preset image clarity), then it can be determined that the confidence level of the face detection result corresponding to the candidate character meets the above-mentioned preset confidence judgment conditions, and the process jumps to execute the subsequent step b2.

[0099] Step b2: If it is determined that the confidence of the face detection result meets the preset confidence judgment condition, based on the candidate facial identity features corresponding to the candidate characters, determine from the character information a character whose facial identity features match the candidate facial identity features as the target character.

[0100] Here, the feature distances (such as Euclidean distance, cosine distance, etc.) between the candidate facial identity features corresponding to the candidate characters and the facial identity features of each character contained in the character information can be calculated respectively, and the character with the smallest feature distance can be determined as the target character that matches the candidate character.

[0101] Step b3: If it is determined that the confidence of the face detection result does not meet the preset confidence judgment condition, based on the candidate human identity features corresponding to the candidate characters, determine from the character information a character whose human identity features match the candidate human identity features as the target character.

[0102] Here, the characteristic distances (such as Euclidean distance, cosine distance, etc.) between the candidate human identity features corresponding to the candidate characters and the human identity features of each character contained in the character information can be calculated respectively, and the character with the smallest characteristic distance can be determined as the target character that matches the candidate characters.

[0103] On the basis of the above steps S101-S103, in order to solve the possible missed detection situation, as an optional embodiment, the method shown in the following step d1 can also be used to perform supplementary detection on the target characters that may be missed in the script video. Specifically:

[0104] Step d1. When the face detection result or body detection result of the target character in the script video is interrupted, if the number of image frames involved in the interruption is less than or equal to the preset number of frames, the face detection result or body detection result of the target character in the script video that is interrupted is supplemented by linear interpolation.

[0105] Specifically, when the face position detection frame (i.e., face detection result) or the body position detection frame (i.e., body detection result) detected for the same target character in the script video is interrupted (i.e., it reappears after a certain number of image frames), if the number of image frames involved in the interruption is small (i.e., less than or equal to the preset number of frames), the face position detection frame (i.e., face detection result) or the body position detection frame (i.e., body detection result) that is interrupted in the script video can be supplemented by linear interpolation.

[0106] It should be noted that the specific value of the above-mentioned preset number of frames can be set according to the actual tracking omission detection requirements, and the embodiments of the present application do not impose any limitation on this.

[0107] On the basis of the above steps S101-S103, in order to solve the possible misdetection situation, as an optional embodiment, the target person who may be tracked incorrectly in the script video can also be corrected according to the method shown in the following steps e1-e2. Specifically:

[0108] Step e1, when the IoU ratio of the face detection result or the body detection result of the target character in the script video between adjacent frames is less than or equal to a first preset IoU threshold, determine whether there is a target revised character in the adjacent frames.

[0109] Specifically, when the IOU (i.e., intersection-over-union) of a face position detection frame (i.e., face detection result) or a body position detection frame (i.e., body detection result) detected in a script video of a target character in the previous frame and the next frame (i.e., between adjacent frames) is less than or equal to a first preset intersection-over-union threshold, it can be determined whether there is a target corrected character whose IOU (i.e., intersection-over-union) with the face position detection frame (i.e., face detection result) or body position detection frame (i.e., body detection result) of other characters appearing in the above-mentioned adjacent frames is greater than or equal to a second preset intersection-over-union threshold (i.e., the intersection-over-union between the target corrected character and the face detection result or body detection result of the target character is greater than or equal to the second preset intersection-over-union threshold). If the above-mentioned target corrected character exists, it indicates that an identity matching error may occur.

[0110] It should be noted that the second preset intersection-and-coupled ratio threshold may be equal to the first preset intersection-and-coupled ratio threshold, or may not be equal to the first preset intersection-and-coupled ratio threshold. The embodiment of the present application does not impose any limitation on the specific values ​​corresponding to the first preset intersection-and-coupled ratio threshold and the second preset intersection-and-coupled ratio threshold.

[0111] Step e2: if it is determined that the target revised character exists in the adjacent frames, the face detection result or body detection result of the target character in the adjacent frames is corrected to the face detection result or body detection result of the target revised character.

[0112] Specifically, referring to the relevant instructions of the above-mentioned step e1, it can be known that if the above-mentioned target correction character exists, it means that the face position detection frame or the body position detection frame that should belong to the above-mentioned target correction character in the above-mentioned adjacent frame is mistakenly matched to the face position detection frame or the body position detection frame of the above-mentioned target character (that is, an identity matching error occurs). Therefore, it is necessary to correct the face detection result or the body detection result of the target character in the adjacent frame (that is, the detection result with a matching error) to the face detection result or the body detection result of the target character.

[0113] On the basis of the above steps S101-S103, if the target tracking method shown in the above steps S101-S103 is applied to a game that allows users to customize and create videos, then as an optional embodiment, the method shown in the following step f1 can also be used to support users to select game characters that play the target characters in the above script video from the game, so as to realize the function of customizing and creating role-playing videos in the game, specifically:

[0114] Step f1, in response to a role-playing instruction for the target character, according to the target character specified in the role-playing instruction to play the target character, the expressions and actions of the target character in the script video are transferred to the target character.

[0115] Here, the above-mentioned role-playing instructions can be control instructions triggered by the user's selection operation on each game character in the game, or they can be control instructions triggered by receiving a screenshot of a game character uploaded by the user; that is, the target character specified in the above-mentioned role-playing instructions to play the target character can be the target game character selected by the user from the game, or it can be the target game character included in the picture uploaded by the user in the game; the specific instruction generation method of the above-mentioned role-playing instructions is not limited in any way in the embodiments of the present application.

[0116] Specifically, for each target character determined in step S103, after determining the target role designated by the user to play the target character according to the above-mentioned role-playing instructions, the expressions and actions of the target character in the script video can be transferred to the above-mentioned target character to obtain a new version of the script video generated after the above-mentioned target role plays the target character.

[0117] On the basis of the above steps S101-S103, in order to meet the actual implementation requirements of the technical solution, when executing the overall service framework of the target tracking method shown in the above steps S101-S103, the execution efficiency of the above overall service framework can be improved from two aspects: model acceleration and inter-frame detection. Specifically:

[0118] In terms of model acceleration, since the execution of the above-mentioned overall service framework involves multiple models (such as face detection model, human body detection model, etc.), in order to save machine and computing costs, an effective way is to accelerate and optimize model reasoning, among which the adopted means is TensorRT model acceleration.

[0119] Here, TensorRT is a high-performance deep learning inference library provided by NVIDIA. It accelerates the inference process of deep learning models through optimization, quantization, and efficient kernel execution. The original model is mainly based on the Pytorch framework for inference. Converting it to a TensorRT model requires the following steps:

[0120] Step 1. This process generally involves two main parts: model transformation and inference.

[0121] Step 2: The conversion part includes: converting the Pytorch model to an ONNX (Open Neural Network Exchange, an open format for representing deep neural network models) model, and then converting the ONNX model to a TensorRT model (e.g., using TensorRT's trtexec tool to convert the ONNX model to the TensorRT engine); ONNX is an open machine learning model format designed to promote interoperability between different deep learning frameworks.

[0122] Step 3: The inference part includes: using the converted TensorRT model for inference, and when the input is consistent, detecting whether the output of the inference pytorch is consistent with the output of TensorRT.

[0123] In terms of inter-frame detection, since video is a medium with a high input information density, the information contained in several consecutive image frames is relatively similar. Therefore, when executing step S103, in addition to frame-by-frame detection, it is also possible to detect once every 2-5 frames, track the target person once, and then use linear interpolation to interpolate the detection frame of the tracked target person (i.e., the face position detection frame or the body position detection frame) for the missing frames in the middle. This inter-frame detection method can save linear multiples of tracking time.

[0124] Based on the target tracking method provided in the embodiment of the present application, the input script video is split into frames to obtain multiple split-frame segments that constitute the script video; face detection and body detection are performed on the multiple split-frame segments respectively, and the information of the characters appearing in the multiple split-frame segments is determined according to the face detection results and body detection results of each split-frame segment; face detection and body detection are performed on each frame image contained in the script video, and the target character appearing in the frame image is determined from the character information according to the face detection results and body detection results of the frame image. In this way, the present application can achieve high-quality, fully automatic human body tracking by combining the advantages of multiple character tracking methods such as face detection and body detection, and is suitable for a variety of application scenarios, effectively overcoming the defect that the prior art is not suitable for processing script-type videos involving multi-lens switching.

[0125] Based on the same inventive concept, the present application also provides a target tracking device corresponding to the above-mentioned target tracking method. Since the principle of solving the problem by the target tracking device in the embodiment of the present application is similar to the above-mentioned target tracking method in the embodiment of the present application, the implementation of the target tracking device can refer to the implementation of the above-mentioned target tracking method, and the repeated parts will not be repeated.

[0126] Reference Figure 4 As shown, Figure 4 A schematic diagram of the structure of a target tracking device provided in an embodiment of the present application is shown, wherein the target tracking device comprises:

[0127] A storyboard module 401 is used to perform storyboard processing on the input script video to obtain a plurality of storyboard segments constituting the script video;

[0128] A detection module 402 is used to perform face detection and body detection on the plurality of storyboard segments, and determine information of characters appearing in the plurality of storyboard segments according to the face detection result and the body detection result of each storyboard segment;

[0129] The matching module 403 is used to perform face detection and body detection on each frame image contained in the script video, and determine the target character contained in the frame image from the character information according to the face detection result and the body detection result of the frame image.

[0130] In an optional implementation, when the input script video is storyboarded, the storyboard module 401 is used to:

[0131] By detecting the pixel difference between adjacent frames in the script video, determining, from the script video, a target adjacent frame whose pixel difference is greater than or equal to a preset pixel difference threshold;

[0132] The script video is segmented by taking each of the target adjacent frames as a video segmentation point to obtain the multiple storyboard segments.

[0133] In an optional implementation, when determining the information of characters in the plurality of storyboard segments according to the face detection result and the body detection result of each storyboard segment, the detection module 402 is configured to:

[0134] For each of the storyboard segments, determine the characters in the storyboard segment according to the face detection result and the body detection result of the storyboard segment; wherein each of the characters corresponds to a set of matching face detection results and body detection results;

[0135] Determine the facial identity features and the human body identity features corresponding to the character according to the face detection results and the human body detection results corresponding to the character, and use the facial identity features and the human body identity features as the character identity information of the character;

[0136] The character identity information of the characters appearing in the multiple storyboard segments is deduplicated to obtain the character identity information of the multiple characters appearing after deduplication as the character information.

[0137] In an optional implementation, when determining the target person included in the frame image from the person information according to the face detection result and the body detection result of the frame image, the matching module 403 is used to:

[0138] Determine candidate characters in the frame image according to the face detection result and the body detection result of the frame image; wherein each candidate character corresponds to a set of matching face detection results and body detection results;

[0139] Determine candidate face identity features and candidate body identity features corresponding to the candidate character according to the face detection results and the body detection results corresponding to the candidate character, and use the candidate face identity features and the candidate body identity features as character identity information of the candidate character;

[0140] From the information of the characters, a character whose character identity information matches the candidate characters is determined as the target character.

[0141] In an optional implementation, when determining, from the character information, a character whose character identity information matches the candidate character as the target character, the matching module 403 is configured to:

[0142] According to the face detection result corresponding to the candidate character, judging whether the confidence of the face detection result meets the preset confidence determination condition;

[0143] If it is determined that the confidence of the face detection result meets the preset confidence determination condition, according to the candidate facial identity features corresponding to the candidate characters, a character whose facial identity features match the candidate facial identity features is determined from the character information as the target character;

[0144] If it is determined that the confidence of the face detection result does not meet the preset confidence judgment condition, based on the candidate human identity features corresponding to the candidate characters, a character whose human identity features match the candidate human identity features is determined from the character information as the target character.

[0145] In an optional implementation, when judging whether the confidence of the face detection result corresponding to the candidate character meets a preset confidence determination condition according to the face detection result, the matching module 403 is used to:

[0146] According to the face detection result corresponding to the candidate person, determining whether the image proportion of the face image of the candidate person in the frame image is greater than or equal to a preset image proportion;

[0147] Under the condition that it is determined that the image proportion of the face image of the candidate person in the frame image is greater than or equal to the preset image proportion, judging whether the image definition of the face image of the candidate person is greater than or equal to the preset image definition;

[0148] If it is determined that the image definition of the facial image of the candidate person is greater than or equal to the preset image definition, it is determined that the confidence of the face detection result corresponding to the candidate person meets the preset confidence judgment condition.

[0149] In an optional implementation, the target tracking device further includes: a missed detection module, wherein the missed detection module is used to:

[0150] When there is an interruption in the face detection result or the body detection result of the target character in the script video, if the number of image frames involved in the interruption is less than or equal to the preset number of frames, the face detection result or the body detection result of the target character in the script video that is interrupted is supplemented by linear interpolation.

[0151] In an optional implementation, the target tracking device further includes: a false detection module, wherein the false detection module is used to:

[0152] When the intersection-and-union ratio of the face detection result or the body detection result of the target person appearing in the script video between adjacent frames is less than or equal to a first preset intersection-and-union ratio threshold, determine whether there is a target revised person appearing in the adjacent frames; wherein the intersection-and-union ratio between the target revised person appearing and the face detection result or the body detection result of the target person appearing is greater than or equal to a second preset intersection-and-union ratio threshold;

[0153] If it is determined that the target revised character exists in the adjacent frame, the face detection result or body detection result of the target character in the adjacent frame is corrected to the face detection result or body detection result of the target revised character.

[0154] In an optional implementation, the target tracking device further includes: a role-playing module, wherein the role-playing module is used to:

[0155] In response to a role-playing instruction for the target character, the expressions and actions of the target character in the script video are transferred to the target character according to the target character specified in the role-playing instruction to play the target character.

[0156] Based on the target tracking device provided by the embodiment of the present application, the input script video is split into frames to obtain multiple split-frame segments that constitute the script video; face detection and body detection are performed on the multiple split-frame segments respectively, and the information of the characters appearing in the multiple split-frame segments is determined according to the face detection results and body detection results of each split-frame segment; face detection and body detection are performed on each frame image contained in the script video, and the target character appearing in the frame image is determined from the character information according to the face detection results and body detection results of the frame image. In this way, the present application can achieve high-quality, fully automatic human body tracking by combining the advantages of multiple character tracking methods such as face detection and body detection, and is suitable for a variety of application scenarios, effectively overcoming the defect that the prior art is not suitable for processing script-type videos involving multi-lens switching.

[0157] Based on the same inventive concept, the present application also provides an electronic device corresponding to the target tracking method. Since the principle of solving the problem by the electronic device in the embodiment of the present application is similar to the above-mentioned target tracking method in the embodiment of the present application, the implementation of the electronic device can refer to the implementation of the above-mentioned target tracking method, and the repeated parts will not be repeated.

[0158] Figure 5A structural diagram of an electronic device 500 provided in an embodiment of the present application includes: a processor 501, a memory 502 and a bus 503, wherein the memory 502 stores machine-readable instructions executable by the processor 501. When the electronic device runs a target tracking method in the embodiment, the processor 501 communicates with the memory 502 via the bus 503, and the processor 501 executes the machine-readable instructions. When the processor 501 executes the machine-readable instructions, the following steps are implemented, specifically:

[0159] Performing storyboard processing on the input script video to obtain a plurality of storyboard segments constituting the script video;

[0160] Performing face detection and body detection on the multiple storyboard segments respectively, and determining information of characters appearing in the multiple storyboard segments according to the face detection result and the body detection result of each storyboard segment;

[0161] Face detection and body detection are performed on each frame image contained in the script video, and according to the face detection result and body detection result of the frame image, a target character contained in the frame image is determined from the character information.

[0162] In an optional implementation, when the input script video is storyboarded, the processor 501 is used to:

[0163] By detecting the pixel difference between adjacent frames in the script video, determining, from the script video, a target adjacent frame whose pixel difference is greater than or equal to a preset pixel difference threshold;

[0164] The script video is segmented by taking each of the target adjacent frames as a video segmentation point to obtain the multiple storyboard segments.

[0165] In an optional implementation, when determining the information of characters in the plurality of storyboard segments according to the face detection result and the body detection result of each storyboard segment, the processor 501 is configured to:

[0166] For each of the storyboard segments, determine the characters in the storyboard segment according to the face detection result and the body detection result of the storyboard segment; wherein each of the characters corresponds to a set of matching face detection results and body detection results;

[0167] Determine the facial identity features and the human body identity features corresponding to the character according to the face detection results and the human body detection results corresponding to the character, and use the facial identity features and the human body identity features as the character identity information of the character;

[0168] The character identity information of the characters appearing in the multiple storyboard segments is deduplicated to obtain the character identity information of the multiple characters appearing after deduplication as the character information.

[0169] In an optional implementation, when determining the target person included in the frame image from the person information according to the face detection result and the body detection result of the frame image, the processor 501 is configured to:

[0170] Determine candidate characters in the frame image according to the face detection result and the body detection result of the frame image; wherein each candidate character corresponds to a set of matching face detection results and body detection results;

[0171] Determine candidate face identity features and candidate body identity features corresponding to the candidate character according to the face detection results and the body detection results corresponding to the candidate character, and use the candidate face identity features and the candidate body identity features as character identity information of the candidate character;

[0172] From the information of the characters, a character whose character identity information matches the candidate characters is determined as the target character.

[0173] In an optional implementation manner, when determining, from the information of the characters, a character whose character identity information matches the candidate character as the target character, the processor 501 is configured to:

[0174] According to the face detection result corresponding to the candidate character, judging whether the confidence of the face detection result meets the preset confidence determination condition;

[0175] If it is determined that the confidence of the face detection result meets the preset confidence determination condition, according to the candidate facial identity features corresponding to the candidate characters, a character whose facial identity features match the candidate facial identity features is determined from the character information as the target character;

[0176] If it is determined that the confidence of the face detection result does not meet the preset confidence judgment condition, based on the candidate human identity features corresponding to the candidate characters, a character whose human identity features match the candidate human identity features is determined from the character information as the target character.

[0177] In an optional implementation, when judging whether the confidence of the face detection result corresponding to the candidate character satisfies a preset confidence determination condition according to the face detection result, the processor 501 is configured to:

[0178] According to the face detection result corresponding to the candidate person, determining whether the image proportion of the face image of the candidate person in the frame image is greater than or equal to a preset image proportion;

[0179] Under the condition that it is determined that the image proportion of the face image of the candidate person in the frame image is greater than or equal to the preset image proportion, judging whether the image definition of the face image of the candidate person is greater than or equal to the preset image definition;

[0180] If it is determined that the image definition of the facial image of the candidate person is greater than or equal to the preset image definition, it is determined that the confidence of the face detection result corresponding to the candidate person meets the preset confidence judgment condition.

[0181] In an optional implementation, the processor 501 is further configured to:

[0182] When there is an interruption in the face detection result or the body detection result of the target character in the script video, if the number of image frames involved in the interruption is less than or equal to the preset number of frames, the face detection result or the body detection result of the target character in the script video that is interrupted is supplemented by linear interpolation.

[0183] In an optional implementation, the processor 501 is further configured to:

[0184] When the intersection-and-union ratio of the face detection result or the body detection result of the target person appearing in the script video between adjacent frames is less than or equal to a first preset intersection-and-union ratio threshold, determine whether there is a target revised person appearing in the adjacent frames; wherein the intersection-and-union ratio between the target revised person appearing and the face detection result or the body detection result of the target person appearing is greater than or equal to a second preset intersection-and-union ratio threshold;

[0185] If it is determined that the target revised character exists in the adjacent frame, the face detection result or body detection result of the target character in the adjacent frame is corrected to the face detection result or body detection result of the target revised character.

[0186] In an optional implementation, the processor 501 is further configured to:

[0187] In response to a role-playing instruction for the target character, the expressions and actions of the target character in the script video are transferred to the target character according to the target character specified in the role-playing instruction to play the target character.

[0188] Through the electronic device provided by the embodiment of the present application, the input script video is split into two parts to obtain multiple split-screen segments that constitute the script video; face detection and body detection are performed on the multiple split-screen segments respectively, and the information of the characters appearing in the multiple split-screen segments is determined according to the face detection result and body detection result of each split-screen segment; face detection and body detection are performed on each frame image contained in the script video, and the target character appearing in the frame image is determined from the character information according to the face detection result and body detection result of the frame image. In this way, the present application can achieve high-quality, fully automatic human body tracking by combining the advantages of multiple character tracking methods such as face detection and body detection, and is suitable for a variety of application scenarios, effectively overcoming the defect that the prior art is not suitable for processing script-type videos involving multi-lens switching.

[0189] Based on the same inventive concept, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. The computer program is executed when a processor is running, and the processor performs the following steps:

[0190] Performing storyboard processing on the input script video to obtain a plurality of storyboard segments constituting the script video;

[0191] Performing face detection and body detection on the multiple storyboard segments respectively, and determining information of characters appearing in the multiple storyboard segments according to the face detection result and the body detection result of each storyboard segment;

[0192] Face detection and body detection are performed on each frame image contained in the script video, and according to the face detection result and body detection result of the frame image, a target character contained in the frame image is determined from the character information.

[0193] In an optional implementation, when the input script video is storyboarded, the processor is used to:

[0194] By detecting the pixel difference between adjacent frames in the script video, determining, from the script video, a target adjacent frame whose pixel difference is greater than or equal to a preset pixel difference threshold;

[0195] The script video is segmented by taking each of the target adjacent frames as a video segmentation point to obtain the multiple storyboard segments.

[0196] In an optional implementation, when determining the information of characters appearing in the plurality of storyboard segments according to the face detection result and the body detection result of each storyboard segment, the processor is configured to:

[0197] For each of the storyboard segments, determine the characters in the storyboard segment according to the face detection result and the body detection result of the storyboard segment; wherein each of the characters corresponds to a set of matching face detection results and body detection results;

[0198] Determine the facial identity features and the human body identity features corresponding to the character according to the face detection results and the human body detection results corresponding to the character, and use the facial identity features and the human body identity features as the character identity information of the character;

[0199] The character identity information of the characters appearing in the multiple storyboard segments is deduplicated to obtain the character identity information of the multiple characters appearing after deduplication as the character information.

[0200] In an optional implementation, when determining the target person included in the frame image from the person information according to the face detection result and the body detection result of the frame image, the processor is configured to:

[0201] Determine candidate characters in the frame image according to the face detection result and the body detection result of the frame image; wherein each candidate character corresponds to a set of matching face detection results and body detection results;

[0202] Determine candidate face identity features and candidate body identity features corresponding to the candidate character according to the face detection results and the body detection results corresponding to the candidate character, and use the candidate face identity features and the candidate body identity features as character identity information of the candidate character;

[0203] From the information of the characters, a character whose character identity information matches the candidate characters is determined as the target character.

[0204] In an optional implementation manner, when determining, from the information of the characters, a character whose character identity information matches the candidate character as the target character, the processor is configured to:

[0205] According to the face detection result corresponding to the candidate character, judging whether the confidence of the face detection result meets the preset confidence determination condition;

[0206] If it is determined that the confidence of the face detection result meets the preset confidence determination condition, according to the candidate facial identity features corresponding to the candidate characters, a character whose facial identity features match the candidate facial identity features is determined from the character information as the target character;

[0207] If it is determined that the confidence of the face detection result does not meet the preset confidence judgment condition, based on the candidate human identity features corresponding to the candidate characters, a character whose human identity features match the candidate human identity features is determined from the character information as the target character.

[0208] In an optional implementation, when judging whether the confidence of the face detection result corresponding to the candidate person meets a preset confidence determination condition based on the face detection result, the processor is configured to:

[0209] According to the face detection result corresponding to the candidate person, determining whether the image proportion of the face image of the candidate person in the frame image is greater than or equal to a preset image proportion;

[0210] Under the condition that it is determined that the image proportion of the face image of the candidate person in the frame image is greater than or equal to the preset image proportion, judging whether the image definition of the face image of the candidate person is greater than or equal to the preset image definition;

[0211] If it is determined that the image definition of the facial image of the candidate person is greater than or equal to the preset image definition, it is determined that the confidence of the face detection result corresponding to the candidate person meets the preset confidence judgment condition.

[0212] In an optional implementation, the processor is further configured to:

[0213] When there is an interruption in the face detection result or the body detection result of the target character in the script video, if the number of image frames involved in the interruption is less than or equal to the preset number of frames, the face detection result or the body detection result of the target character in the script video that is interrupted is supplemented by linear interpolation.

[0214] In an optional implementation, the processor is further configured to:

[0215] When the intersection-and-union ratio of the face detection result or the body detection result of the target person appearing in the script video between adjacent frames is less than or equal to a first preset intersection-and-union ratio threshold, determine whether there is a target revised person appearing in the adjacent frames; wherein the intersection-and-union ratio between the target revised person appearing and the face detection result or the body detection result of the target person appearing is greater than or equal to a second preset intersection-and-union ratio threshold;

[0216] If it is determined that the target revised character exists in the adjacent frame, the face detection result or body detection result of the target character in the adjacent frame is corrected to the face detection result or body detection result of the target revised character.

[0217] In an optional implementation, the processor is further configured to:

[0218] In response to a role-playing instruction for the target character, the expressions and actions of the target character in the script video are transferred to the target character according to the target character specified in the role-playing instruction to play the target character.

[0219] Through the above-mentioned computer-readable storage medium provided by the embodiment of the present application, the input script video is split into frames to obtain multiple split-frame segments that constitute the script video; face detection and body detection are performed on the multiple split-frame segments respectively, and the information of the characters appearing in the multiple split-frame segments is determined according to the face detection results and body detection results of each split-frame segment; face detection and body detection are performed on each frame image contained in the script video, and the target character appearing in the frame image is determined from the character information according to the face detection results and body detection results of the frame image. In this way, the present application can achieve high-quality, fully automatic human body tracking by combining the advantages of multiple character tracking methods such as face detection and body detection, and is suitable for a variety of application scenarios, effectively overcoming the defect that the prior art is not suitable for processing script-type videos involving multi-lens switching.

[0220] In an embodiment of the present application, the computer-readable storage medium can also execute other machine-readable instructions when run by the processor to execute the target tracking method as described in other embodiments. For the specific steps and principles of the target tracking method, please refer to the description of the method side embodiment, which will not be repeated here.

[0221] In the embodiments provided in the present application, it should be understood that the disclosed systems and methods can be implemented in other ways. The system embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of systems or units, which can be electrical, mechanical or other forms.

[0222] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0223] In addition, each functional unit in the embodiments provided in the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0224] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0225] It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description and are not to be understood as indicating or implying relative importance.

[0226] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the above-mentioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-mentioned embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application. They should all be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A target tracking method, characterized in that: The target tracking method comprises: Performing storyboard processing on the input script video to obtain a plurality of storyboard segments constituting the script video; Performing face detection and body detection on the multiple storyboard segments respectively, and determining information of characters appearing in the multiple storyboard segments according to the face detection result and the body detection result of each storyboard segment; Face detection and body detection are performed on each frame image contained in the script video, and according to the face detection result and body detection result of the frame image, a target character contained in the frame image is determined from the character information.

2. The target tracking method according to claim 1, characterized in that: The storyboard processing of the input script video includes: By detecting the pixel difference between adjacent frames in the script video, determining, from the script video, a target adjacent frame whose pixel difference is greater than or equal to a preset pixel difference threshold; The script video is segmented by taking each of the target adjacent frames as a video segmentation point to obtain the multiple storyboard segments.

3. The target tracking method according to claim 1, characterized in that: The step of determining information of characters appearing in the plurality of storyboard segments according to the face detection result and the body detection result of each storyboard segment comprises: For each of the storyboard segments, determine the characters in the storyboard segment according to the face detection result and the body detection result of the storyboard segment; wherein each of the characters corresponds to a set of matching face detection results and body detection results; Determine the facial identity features and the human body identity features corresponding to the character according to the face detection results and the human body detection results corresponding to the character, and use the facial identity features and the human body identity features as the character identity information of the character; The character identity information of the characters appearing in the multiple storyboard segments is deduplicated to obtain the character identity information of the multiple characters appearing after deduplication as the character information.

4. The target tracking method according to claim 1, characterized in that: The step of determining the target person included in the frame image from the person information according to the face detection result and the body detection result of the frame image comprises: Determine candidate characters in the frame image according to the face detection result and the body detection result of the frame image; wherein each candidate character corresponds to a set of matching face detection results and body detection results; Determine candidate face identity features and candidate body identity features corresponding to the candidate character according to the face detection results and the body detection results corresponding to the candidate character, and use the candidate face identity features and the candidate body identity features as character identity information of the candidate character; From the information of the characters, a character whose character identity information matches the candidate characters is determined as the target character.

5. The target tracking method according to claim 4, characterized in that: The step of determining, from the information of the characters, a character whose character identity information matches the candidate characters as the target character, includes: According to the face detection result corresponding to the candidate character, judging whether the confidence of the face detection result meets the preset confidence determination condition; If it is determined that the confidence of the face detection result meets the preset confidence determination condition, according to the candidate facial identity features corresponding to the candidate characters, a character whose facial identity features match the candidate facial identity features is determined from the character information as the target character; If it is determined that the confidence of the face detection result does not meet the preset confidence judgment condition, based on the candidate human identity features corresponding to the candidate characters, a character whose human identity features match the candidate human identity features is determined from the character information as the target character.

6. The target tracking method according to claim 5, characterized in that: The step of judging whether the confidence of the face detection result corresponding to the candidate character meets a preset confidence judgment condition according to the face detection result corresponding to the candidate character includes: According to the face detection result corresponding to the candidate person, determining whether the image proportion of the face image of the candidate person in the frame image is greater than or equal to a preset image proportion; Under the condition that it is determined that the image proportion of the face image of the candidate person in the frame image is greater than or equal to the preset image proportion, judging whether the image definition of the face image of the candidate person is greater than or equal to the preset image definition; If it is determined that the image definition of the facial image of the candidate person is greater than or equal to the preset image definition, it is determined that the confidence of the face detection result corresponding to the candidate person meets the preset confidence judgment condition.

7. The target tracking method according to claim 1, characterized in that: The target tracking method further includes: When there is an interruption in the face detection result or the body detection result of the target character in the script video, if the number of image frames involved in the interruption is less than or equal to the preset number of frames, the face detection result or the body detection result of the target character in the script video that is interrupted is supplemented by linear interpolation.

8. The target tracking method according to claim 1, characterized in that: The target tracking method further comprises: When the intersection-and-union ratio of the face detection result or the body detection result of the target person appearing in the script video between adjacent frames is less than or equal to a first preset intersection-and-union ratio threshold, determine whether there is a target revised person appearing in the adjacent frames; wherein the intersection-and-union ratio between the target revised person appearing and the face detection result or the body detection result of the target person appearing is greater than or equal to a second preset intersection-and-union ratio threshold; If it is determined that the target revised character exists in the adjacent frame, the face detection result or body detection result of the target character in the adjacent frame is corrected to the face detection result or body detection result of the target revised character.

9. The target tracking method according to claim 1, characterized in that: The target tracking method further comprises: In response to a role-playing instruction for the target character, the expressions and actions of the target character in the script video are transferred to the target character according to the target character specified in the role-playing instruction to play the target character.

10. A target tracking device, characterized in that: The target tracking device comprises: A storyboard module, used for storyboarding an input script video to obtain a plurality of storyboard segments constituting the script video; A detection module, configured to perform face detection and body detection on the plurality of storyboard segments respectively, and determine information of characters appearing in the plurality of storyboard segments according to the face detection result and the body detection result of each storyboard segment; The matching module is used to perform face detection and body detection on each frame image contained in the script video, and determine the target character contained in the frame image from the character information according to the face detection result and the body detection result of the frame image.

11. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the target tracking method as described in any one of claims 1 to 9 are performed.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the target tracking method according to any one of claims 1 to 9 are executed.

Images