Audio and video segmentation method and device, electronic equipment and storage medium

Abstract

The invention discloses an audio and video segmentation method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a target video and a target audio, and enabling the target video and the target audio to be obtained through the collection of a target scene; tracking the target object by using the target video to obtain tracking data; and segmenting an object audio and an object video belonging to the target object from the target audio and the target video by using the tracking data. By means of the mode, the audio and video segmentation accuracy can be improved.

Description

Technical Field

[0001] The present application relates to the field of audio and video technology, and in particular to an audio and video segmentation method, device, electronic device and storage medium. Background Art

[0002] In some target scenarios (such as government halls, business halls, bank halls, high-speed rail security checkpoints, aircraft security checkpoints, etc.), video acquisition devices (such as cameras) continuously collect videos of staff and passengers, and audio acquisition devices (such as microphones) continuously collect audio of staff and passengers. In the future, each passenger needs to be analyzed one by one, so it is necessary to accurately determine the audio and video corresponding to each passenger from the continuously collected video and audio, so as to accurately analyze the passengers. Summary of the invention

[0003] The main technical problem solved by the present application is to provide an audio and video segmentation method, device, electronic device and storage medium, which can improve the accuracy of audio and video segmentation.

[0004] To solve the above technical problems, a technical solution adopted in the present application is: to provide an audio and video segmentation method, the method comprising: obtaining a target video and a target audio, wherein the target video and the target audio are obtained by collecting a target scene; using the target video, tracking the target object to obtain tracking data; using the tracking data, segmenting the object audio and the object video belonging to the target object from the target audio and the target video.

[0005] The tracking data includes the tracking end time; and the tracking end determination step includes: in response to failure of tracking of the target object, determining that tracking of the target object is ended.

[0006] Among them, the failure of tracking the target object includes: in response to the loss of tracking of the target object, obtaining at least two target video frames after the tracking is lost; for each target video frame, obtaining the similarity between the target video frame and any historical video frame before the tracking is lost; based on the similarities corresponding to each target video frame, determining the failure of tracking the target object.

[0007] The step of obtaining at least two target video frames after the tracking is lost includes: obtaining a number of video frames after the tracking is lost; and selecting a video frame whose quality meets the quality requirement from the number of video frames as the target video frame.

[0008] Among them, based on the similarity corresponding to each target video frame, determining the failure of tracking the target object includes: for each target video frame, converting the similarity corresponding to the target video frame to obtain a result representation value; and comprehensively combining the result representation values ​​corresponding to each target video frame to determine the failure of tracking the target object.

[0009] Among them, the similarity corresponding to the target video frame is converted to obtain a result representation value, including: in response to the similarity corresponding to the target video frame being greater than or equal to a similarity threshold, the similarity corresponding to the target video frame is converted into a first result representation value; in response to the similarity corresponding to the target video frame being less than the similarity threshold, the similarity corresponding to the target video frame is converted into a second result representation value.

[0010] Among them, the first result characterization value and the second result characterization value are opposite in sign and equal in value, and the number of target video frames is an odd number; comprehensively analyzing the result characterization values ​​corresponding to each target video frame, determining that tracking of the target object is invalid, including: summing the result characterization values ​​corresponding to each target video frame; in response to the sum being less than a preset value, determining that tracking of the target object is invalid.

[0011] The audio and video segmentation method further includes: in response to the sum being greater than a preset value, determining to continue tracking the target object.

[0012] The tracking data includes tracking moments, and the tracking moments include tracking start moments, tracking middle moments and tracking end moments, and the tracking middle moments include moments when tracking of the target object is lost and moments when tracking of the target object continues; using the tracking data, object audio and object video belonging to the target object are cut out from the target audio and target video, including: in response to obtaining any tracking moment other than the start moment, as the first target moment; using the target time interval, the target audio and target video are cut to obtain target audio segments and target video segments; wherein the target time interval includes the second target moment, the first target moment and the moment between the first target moment and the second target moment, and the second target moment is the tracking moment before the first target moment; the video cutting method also includes: in response to the end of tracking of the target object, the target audio segments belonging to the target object are merged to obtain the object audio of the corresponding target object; and the target video segments belonging to the target object are merged to obtain the object video of the corresponding target object.

[0013] The target video is captured by a video capture device, the target object is an object in the target scene whose distance from the video capture device meets a preset distance requirement; and / or the target object includes a human face.

[0014] To solve the above technical problems, another technical solution adopted in the present application is: to provide an audio and video segmentation device, which includes an acquisition module, a tracking module and a segmentation module; the acquisition module is used to acquire target video and target audio, wherein the target video and target audio are obtained by collecting the target scene; the tracking module is used to use the target video to track the target object and obtain tracking data; the segmentation module is used to use the tracking data to segment the object audio and object video belonging to the target object from the target audio and target video.

[0015] In order to solve the above technical problems, another technical solution adopted in the present application is: to provide an electronic device, the electronic device includes a memory and a processor, the memory stores program instructions, and the program instructions can be executed to implement the above method.

[0016] In order to solve the above technical problem, another technical solution adopted by the present application is: providing a computer-readable storage medium, which is used to store program instructions, and the program instructions can be executed to implement the above method.

[0017] In the above technical solution, when the target object can be tracked, it indicates that the target object is completing the pending task in the target scene, and the video collected by the video acquisition device is the video about the target object, and the audio collected by the audio acquisition device is the audio about the target object. Therefore, the tracking data obtained by tracking the target object can assist in segmenting the audio and video belonging to the target object from the target audio and target video collected from the target scene, thereby improving the convenience and accuracy of audio and video segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a flowchart of an embodiment of an audio and video segmentation method provided by the present application;

[0019] Figure 2 It is a flowchart of an embodiment of a method for determining failure of tracking of a target object provided by the present application;

[0020] Figure 3 yes Figure 2 The flowchart of step S23 is shown as an embodiment;

[0021] Figure 4 It is a structural schematic diagram of an embodiment of an audio and video segmentation device provided by the present application;

[0022] Figure 5 It is a structural schematic diagram of an embodiment of an electronic device provided by the present application;

[0023] Figure 6 It is a structural schematic diagram of an embodiment of a computer-readable storage medium provided by the present application. DETAILED DESCRIPTION

[0024] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0025] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0026] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the objects associated before and after are in an "or" relationship. In addition, "many" in this article means two or more than two. In addition, the term "at least one" in this article means any combination of at least two of any one or more of a plurality of, for example, including at least one of A, B, and C, can mean including any one or more elements selected from the set consisting of A, B, and C.

[0027] See also Figure 1 , Figure 1 1 is a flow chart of an embodiment of the audio and video segmentation method provided by the present application. It should be noted that if there are substantially the same results, this embodiment does not necessarily use Figure 1 The process sequence shown is limited. Figure 1 As shown, this embodiment includes:

[0028] Step S11: Acquire target video and target audio.

[0029] In some target scenarios (such as government halls, business halls, bank halls, high-speed rail security checkpoints, aircraft security checkpoints, etc.), video acquisition devices (such as cameras) continuously collect videos of staff and customers, and audio acquisition devices (such as microphones) continuously collect audio of staff and customers. In the future, it is necessary to analyze each customer one by one, so it is necessary to determine the audio and video corresponding to each customer from the continuously collected video and audio, so as to accurately analyze the customer.

[0030] In this embodiment, the target video and the target audio are obtained, wherein the target video and the target audio are obtained by collecting the target scene. It should be noted that, as described above, the target video is obtained by collecting the target scene by the video collection device, and the target audio is obtained by collecting the target scene by the audio collection device. The target scene can be a government hall, a business hall, a bank hall, a high-speed rail security checkpoint, an aircraft security checkpoint, etc., which is not limited here.

[0031] Step S12: using the target video, tracking the target object to obtain tracking data.

[0032] In this embodiment, the target video is used to track the target object to obtain tracking data. Specifically, the target video is analyzed using tracking technology to determine the tracking object - the target object, and the determined tracking object - the target object is tracked, wherein the tracking technology is not limited and can be specifically set according to actual use needs.

[0033] If the target object can be tracked, it indicates that the target object is completing the pending task in the target scene. For example, in the high-speed rail security inspection scene, if the target object can be tracked, it indicates that the target object is being inspected by the security inspector; for example, in the bank hall scene, if the target object can be tracked, it indicates that the target object is handling banking business (such as savings card business, credit card business, loan business, insurance business, etc.). When the target object is completing the pending task in the target scene, the video captured by the video acquisition device is the video of the target object, and the audio captured by the audio acquisition device is the audio of the target object.

[0034] If the target object has completed the pending task in the target scene, the video data of the target object cannot be collected, and the target object cannot be tracked further. Therefore, when the target object cannot be tracked, the video collected by the video collection device is not the video of the target object, and the video collected by the audio collection device is not the audio of the target object.

[0035] Therefore, the video of the target object collected by the video acquisition device and the audio of the target object collected by the audio acquisition device can be determined by tracking the target object. That is, the tracking data obtained by tracking the target object can help to separate the audio and video belonging to the target object from the target audio and target video collected from the target scene.

[0036] It should be noted that the tracking of the target object using the target video is continuous tracking. The current video frame will be continuously compared with the historical video frame to determine whether the object in the current video frame and the historical video frame is the same object, so as to be able to promptly determine whether the tracked object has changed, so that the audio and video belonging to the target object can be accurately segmented later.

[0037] In addition, after the tracking of a target object is completed, the next target object will be waited for to enter and tracked, and the audio and video belonging to the next target object will be segmented out later, and the cycle will be repeated until the user turns off the video acquisition device and stops the cycle. For example, at the high-speed rail security checkpoint, the security check of passengers is carried out one by one from the front to the back; therefore, the passenger at the front of the line, that is, the passenger closest to the video acquisition device among all passengers, is the passenger who is being checked by the security inspector or is about to be checked by the security inspector, and is regarded as the target object A; after the target object A leaves after the security check, the video data of the target object A cannot be collected, and the target object A cannot be tracked further, and the tracking of the target object A ends; after the target object B behind the target object A enters, the target object B is continuously tracked, and the audio and video belonging to the target object B can be segmented out later.

[0038] In one embodiment, the target object is a human face, that is, in the present application, the human face is continuously tracked, and the tracking data obtained by tracking the human face is used to assist in segmenting the audio and video belonging to the human face from the target audio and target video collected from the target scene. In the case where the target object is a human face, different objects can be better distinguished to more accurately determine whether the tracked object has changed, so that the audio and video belonging to different objects can be accurately segmented later.

[0039] Of course, in other implementations, the target object may also be the upper body of the object, the head of the object, the legs of the object, etc., which is not limited here.

[0040] In one embodiment, the target video is collected by a video acquisition device, and the target object is an object in the target scene whose distance from the video acquisition device meets the preset distance requirement. The preset distance requirement is not limited and can be specifically set according to actual use needs. For example, the preset distance requirement is the minimum distance among the distances corresponding to all objects in the target scene; when the preset distance requirement is the minimum distance among the distances corresponding to all objects in the target scene, it indicates that the target object is the object closest to the video acquisition device in the target scene, and the object closest to the video acquisition device is the object currently being collected by the video acquisition device, so the object currently being collected by the audio acquisition device is also the target object.

[0041] For example, take the target scene as a high-speed rail security checkpoint and the preset distance requirement as the minimum distance among the distances corresponding to all objects in the target scene: at the high-speed rail security checkpoint, passengers are checked one by one from the front to the back; therefore, the passenger at the front of the line, that is, the passenger closest to the video acquisition device among all passengers, who is undergoing or about to undergo security inspection by the security inspector, is taken as the target object.

[0042] Furthermore, after target object A leaves after security check, the video data of target object A cannot be collected, and target object A cannot be tracked any further, and the tracking of target object A ends; after target object B, which is behind target object A, enters, the distance is the minimum among the distances corresponding to all objects in the target scene, so target object B is used as the new target object, and target object B is continuously tracked, and the audio and video belonging to target object B can be segmented out later.

[0043] In one embodiment, the tracking data includes but is not limited to the tracking start time, the tracking end time, the tracking intermediate time, etc., which are not limited here. The tracking start time is the time when the target object is first detected and tracked. The tracking end time is the time when the tracking of the target object is ended. The tracking intermediate time is the tracking time between the tracking start time and the tracking end time.

[0044] In a specific implementation, the intermediate tracking moments include, but are not limited to, moments when the tracking of the target object is lost and moments when the tracking of the target object is continued.

[0045] Step S13: Using the tracking data, segment the target audio and target video belonging to the target object from the target audio and target video.

[0046] In this embodiment, the tracking data is used to cut out the object audio and object video belonging to the target object from the target video and target audio. When the target object can be tracked, it indicates that the target object is completing the pending task in the target scene, and the video corresponding to the video acquisition device is the video about the target object, and the audio collected by the audio acquisition device is the audio about the target object. Therefore, the tracking data obtained by tracking the target object can assist in cutting out the audio and video belonging to the target object from the target audio and target video obtained by collecting the target scene, thereby improving the convenience and accuracy of audio and video cutting. It should be noted that cutting out the object audio belonging to the target object from the target audio can be understood as caching the audio corresponding to the target object from the target audio; cutting out the object video belonging to the target object from the target video can be understood as caching the video corresponding to the target object from the target video.

[0047] In one embodiment, the tracking data includes the tracking end time, and the tracking end determination step includes: in response to the failure of tracking the target object, determining that the tracking of the target object is ended. That is, in the case of failure of tracking the target object, it is considered that the target object cannot be tracked any further, the tracking of the target object is ended, and after waiting for the next target object to enter, the next target object is continuously tracked.

[0048] In a specific embodiment, the target object tracking failure is specifically: in response to the target object tracking loss, it is determined that the target object tracking failure. That is, in the case of target object tracking loss, that is, in the case of the target object being unable to be tracked at present, the target object tracking ends.

[0049] The tracking of the target object may be temporarily lost. For example, if the target object is a human face, when the person lowers his head, bends over to pick up something, or turns his head to talk to the person behind him, the video data collected by the video acquisition device may temporarily not contain the human face, and the tracking of the human face is temporarily lost. For another example, if the target object is a human face, when the glasses on the face are taken off and wiped, the face changes, and it may be erroneously determined that the tracking of the face is lost. Therefore, if the tracking of the target object is temporarily lost, it is directly considered that the target object cannot be tracked anymore, and the tracking of the target object is invalid, which will cause abnormal tracking of the target object. Based on the tracking data obtained by tracking the target object, the audio and video belonging to the target object obtained by segmentation are inaccurate.

[0050] Therefore, in other specific embodiments, Figure 2 As shown, Figure 2 This is a flow chart of an embodiment of a method for determining failure of tracking of a target object provided by the present application. Failure of tracking of a target object specifically includes the following sub-steps:

[0051] Step S21: in response to the loss of tracking of the target object, obtaining at least two target video frames after the tracking is lost.

[0052] After determining that the tracking of the target object is lost, it is necessary to further compare the video frames after the tracking of the target object is lost with the video frames before the tracking is lost to accurately determine whether the tracking of the target object is temporarily lost, thereby determining whether the tracking of the target object is invalid.

[0053] Therefore, in this embodiment, in response to the loss of tracking of the target object, at least two target video frames after the tracking is lost are obtained. The number of video frames obtained after the tracking is lost is not limited, for example, 2 target video frames, 3 target video frames, 5 target video frames, 10 target video frames, etc.

[0054] In one embodiment, obtaining at least two target video frames after tracking is lost is specifically: obtaining several video frames after tracking is lost; selecting video frames whose quality meets the quality requirements from the several video frames as target video frames. In other words, selecting high-quality video frames from the video frames after tracking is lost as target video frames; using the high-quality video frames after tracking is lost, it is possible to accurately determine whether the tracking of the target object is temporarily lost, thereby accurately determining whether the tracking of the target object is invalid.

[0055] There is no limitation on the quality requirement. For example, the quality detection network is used to detect the quality of the video frame, and the quality requirement may be a quality greater than or equal to 0.8. In addition, there is no limitation on the number of video frames acquired after the tracking is lost, for example, 5, 10, 15, 20, etc. Furthermore, the acquired video frames after the tracking is lost may be the interval video frames after the tracking is lost, or may be the adjacent video frames.

[0056] In other implementations, the at least two target video frames obtained after the tracking is lost may also be any two video frames after the tracking is lost.

[0057] In one embodiment, the at least two target video frames obtained after tracking is lost may be video frames at adjacent positions or video frames at alternate positions.

[0058] Step S22: For each target video frame, obtain the similarity between the target video frame and any historical video frame before tracking is lost.

[0059] In this embodiment, for each target video frame, the similarity between the target video frame and any historical video frame before the tracking is lost is obtained. For each target video frame after the tracking is lost, the target video frame after the tracking is lost is compared with the historical video frame before the tracking is lost to determine the similarity between the target video frame after the tracking is lost and the historical video frame before the tracking is lost. Subsequently, based on the similarity between the target video frame after the tracking is lost and the historical video frame before the tracking is lost, it can be determined whether the tracking of the target object is temporarily lost, thereby determining whether the tracking of the target object is invalid.

[0060] Step S23: Based on the similarities corresponding to the target video frames, it is determined that the tracking of the target object is invalid.

[0061] In this embodiment, based on the similarity corresponding to each target video frame, it is determined that the tracking of the target object is invalid. That is to say, by comprehensively analyzing the similarity between each target video frame after the tracking is lost and the historical video frame before the tracking is lost, it is determined whether the tracking of the target object is temporarily lost, thereby determining whether the tracking of the target object is invalid. The similarity between a single target video frame after the tracking is lost and the historical video frame before the tracking is lost is accidental, so by comprehensively analyzing multiple target video frames after the tracking is lost, it is possible to accurately determine whether the tracking of the target object is temporarily lost, thereby accurately determining whether the tracking of the target object is invalid.

[0062] In one embodiment, whether the tracking of the target object is invalid can be determined by the proportion of the number of target video frames whose similarity is greater than the similarity threshold. Among them, when the proportion of the number of target video frames whose similarity is greater than the similarity threshold is greater than the preset proportion, it means that the target object can be tracked most of the time when the tracking of the target object is lost, so the loss of tracking of the target object is temporary, and the tracking of the target object is determined to be effective, and the target object is continued to be tracked. When the proportion of the number of target video frames whose similarity is greater than the similarity threshold is less than or equal to the preset proportion, it means that the target object can be tracked in a small part of the time when the tracking of the target object is lost, which can be almost ignored, so the loss of tracking of the target object is not temporary, and the tracking of the target object is determined to be invalid, and the tracking of the target object is terminated.

[0063] There is no limitation on the preset proportion and similarity threshold, and they can be set according to actual needs.

[0064] In a specific embodiment, Figure 3 As shown, Figure 3 yes Figure 2 The flowchart of step S23 of an embodiment shown in FIG. 1 is to determine the failure of tracking the target object based on the similarity corresponding to each target video frame, and specifically includes the following sub-steps:

[0065] Step S31: for each target video frame, convert the similarity corresponding to the target video frame to obtain a result representation value.

[0066] In this embodiment, for each target video frame, the similarity corresponding to the target video frame is converted to obtain a result representation value. That is, before using the similarity corresponding to each target video frame to determine whether the tracking of the target object is invalid, the similarity corresponding to the target video frame is converted to determine whether the tracking of the target object is invalid based on the result representation value obtained by converting the similarity corresponding to the target video frame. By simplifying the result representation value of the similarity of the target video frame, it is possible to determine whether the tracking of the target object is invalid more quickly later.

[0067] In one embodiment, in response to the similarity corresponding to the target video frame being greater than or equal to the similarity threshold, the similarity corresponding to the target video frame is converted into a first result representation value. That is, when the similarity corresponding to the target video frame is greater than or equal to the similarity threshold, the similarity corresponding to the target video frame is converted into a first result representation value.

[0068] The first result characterization value is not limited and can be set according to actual use needs. For example, the first result characterization value is 1, 2, 3, 5 or 10. For another example, the first result characterization value is a, b or c. In addition, the similarity threshold is not limited and can be set according to actual use needs.

[0069] In one embodiment, in response to the similarity corresponding to the target video frame being less than the similarity threshold, the similarity corresponding to the target video frame is converted into a second result representation value. In other words, when the similarity corresponding to the target video frame is less than the similarity threshold, the similarity corresponding to the target video frame is converted into a second result representation value.

[0070] The second result characterization value is not limited and can be set according to actual use needs. For example, the second result characterization value is -1, -2, -3, -5 or -10. For another example, the first result characterization value is -a, -b or -c.

[0071] Step S32: synthesize the result representation values ​​corresponding to each target video frame to determine whether tracking of the target object is invalid.

[0072] In this embodiment, the result representation values ​​corresponding to each target video frame are combined to determine whether the tracking of the target object is invalid. The similarity between a single target video frame after the tracking is lost and the historical video frame before the tracking is lost is accidental, that is, the result representation value corresponding to the single target video frame after the tracking is lost is accidental; therefore, by combining the result representation values ​​corresponding to multiple target video frames after the tracking is lost, it is possible to accurately determine whether the tracking of the target object is temporarily lost, and thus it is possible to accurately determine whether the tracking of the target object is invalid.

[0073] In one embodiment, when the similarity corresponding to the target video frame is greater than or equal to the similarity threshold, the similarity corresponding to the target video frame is converted into a first result representation value, and when the similarity corresponding to the target video frame is less than the similarity threshold, the similarity corresponding to the target video frame is converted into a second result representation value; the first result representation value and the second result representation value are opposite in sign and equal in value, and the number of target video frames is an odd number; at this time, the result representation values ​​corresponding to each target video frame are comprehensively considered to determine that the tracking of the target object is invalid, specifically: the result representation values ​​corresponding to each target video frame are summed; in response to the sum being less than a preset value, it is determined that the tracking of the target object is invalid.

[0074] Since the first result characterization value and the second result characterization value are opposite in sign and equal in value, when the result characterization values ​​corresponding to each target video frame are summed, the first result characterization value and the second result characterization value will cancel each other out. However, since the target video frame is odd in number, the first result characterization value and the second result characterization value will not completely cancel each other out, and there will inevitably be a result characterization value remaining. When the sum of the result characterization values ​​corresponding to each target video frame is less than the preset value, it indicates that the remaining result characterization value is a negative result characterization value. At this point, it is determined that the tracking of the target object is not temporarily lost, and thus it is determined that the tracking of the target object has failed, that is, the tracking of the target object is ended.

[0075] It should be noted that the idea of ​​the first result characterization value and the second result characterization value offsetting each other can be understood as the idea of ​​offsetting the negative votes (second result characterization value) and the positive votes (first result characterization value) to sum up the negative votes and positive votes to obtain the final voting result.

[0076] In other embodiments, when the similarity corresponding to the target video frame is greater than or equal to the similarity threshold, the similarity corresponding to the target video frame is converted into a first result representation value; when the similarity corresponding to the target video frame is less than the similarity threshold, the similarity corresponding to the target video frame is converted into a second result representation value; the first result representation value and the second result representation value are opposite in sign and equal in value, and the number of target video frames is an odd number; at this time, the result representation values ​​corresponding to each target video frame are summed; in response to the sum being greater than a preset value, it is determined that the tracking of the target object continues.

[0077] Since the first result characterization value and the second result characterization value are opposite in sign and equal in value, the result characterization values ​​corresponding to each target video frame are summed, and the first result characterization value and the second result characterization value will cancel each other out. However, since the target video frame is odd in number, the first result characterization value and the second result characterization value will not completely cancel each other out, and there will inevitably be a result characterization value remaining. When the sum of the result characterization values ​​corresponding to each target video frame is greater than the preset value, it indicates that the remaining result characterization value is a positive result characterization value. At this point, it is determined that the tracking of the target object is temporarily lost, and thus it is determined that the tracking of the target object is valid, that is, the tracking of the target object continues.

[0078] In one embodiment, when the similarity corresponding to the target video frame is greater than or equal to the similarity threshold, the similarity corresponding to the target video frame is converted into a first result characterization value, and when the similarity corresponding to the target video frame is less than the similarity threshold, the similarity corresponding to the target video frame is converted into a second result characterization value; the first result characterization value and the second result characterization value are opposite in sign and equal in value, and the number of target video frames is an odd number; at this time, the number of the first result characterization values ​​and the second result characterization values ​​can also be counted, and when the number of the first result characterization values ​​is less than the number of the second result characterization values, it is determined that the tracking of the target object is not temporarily lost, thereby determining that the tracking of the target object is invalid, that is, the tracking of the target object ends; and when the number of the first result characterization values ​​is greater than the number of the second result characterization values, it is determined that the tracking of the target object is temporarily lost, thereby determining that the tracking of the target object is valid, that is, the tracking of the target object continues.

[0079] It should be noted that counting the number of first result characterization values ​​and second result characterization values ​​can be understood as counting the number of negative votes (second result characterization values) and positive votes (first result characterization values).

[0080] In one embodiment, the tracking data includes the tracking start time and the tracking end time, and the tracking data is used to segment the target audio and the target video belonging to the target object from the target audio and the target video, specifically: using the target time interval, the target audio and the target video are segmented to obtain the target audio and the target video; wherein the target time interval includes the tracking start time, the tracking end time, and the time between the tracking start time and the tracking end time. That is, after determining the tracking end time of the target object, the target audio and the target video belonging to the target object are segmented from the target audio and the target video as a whole. That is, before determining the tracking end time, the audio and video are not segmented in the middle.

[0081] In other embodiments, the tracking data includes tracking moments, and the tracking moments include tracking start moments, tracking middle moments, and tracking end moments, and the tracking middle moments include moments when tracking of the target object is lost and moments when tracking of the target object continues; using the tracking data, object audio and object video belonging to the target object are segmented from the target audio and target video, specifically: in response to obtaining any tracking moment other than the start moment, as the first target moment; using the target time interval, the target audio and target video are segmented to obtain target audio segments and target video segments; wherein the target time interval includes the second target moment, the first target moment, and the moment between the first target moment and the second target moment, and the second target moment is the tracking moment before the first target moment.

[0082] After the tracking of the target object begins, the target object is tracked continuously; because it is uncertain whether the loss of tracking of the target object is temporary or non-temporary, after it is determined that the tracking of the target object is lost, the target time interval A composed of the moment when the tracking of the target object is lost and the moment when the tracking starts is determined, the content corresponding to the target time interval A in the target audio is cached, and the content corresponding to the target time interval A in the target video is cached to obtain the target audio segment and the target video segment corresponding to the target time interval A.

[0083] If it is determined that the loss of tracking of the target object is temporary, then the audio and video collected during the time interval for determining whether the loss of tracking of the target object is temporary are also audio and video about the target object. Therefore, the target time interval B formed by the moment when the tracking of the target object is lost and the continuation of tracking of the target object is determined, the content corresponding to the target time interval B in the target audio is cached, and the content corresponding to the target time interval B in the target video is cached to obtain the target audio segment and target video segment corresponding to the target time interval B. When it is determined that the tracking of the target object is continued, the target object is continuously tracked until it is determined again that the tracking of the target object is lost and the loss of tracking of the target object is non-temporary, and it is determined that the tracking of the target object is invalid, and the tracking of the target object is terminated; at this time, the moment when the tracking of the target object is lost is used as the tracking end moment, and the target time interval C formed by the tracking end moment and the target object tracking continuation moment is determined, the content corresponding to the target time interval C in the target audio is cached, and the content corresponding to the target time interval C in the target video is cached to obtain the target audio segment and target video segment corresponding to the target time interval C.

[0084] If it is determined that the loss of tracking of the target object is non-temporary, then the tracking of the target object is determined to be invalid and the tracking of the target object is terminated. Therefore, the moment when the tracking of the target object is lost is used as the tracking end moment, the target time interval D composed of the tracking end moment and the tracking start moment is determined, the content corresponding to the target time interval D in the target audio is cached, and the content corresponding to the target time interval D in the target video is cached, and the target audio segment and the target video segment corresponding to the target time interval D are obtained.

[0085] Furthermore, since the target audio segments and video segments belonging to the target object are segmented, the target audio segments and video segments belonging to the target object need to be merged to form a one-person-one-file record. Specifically, in response to the end of tracking the target object, the target audio segments belonging to the target object are merged to obtain the object audio of the corresponding target object; and the target video segments belonging to the target object are merged to obtain the object video of the corresponding target object.

[0086] In one implementation, the video clips belonging to the target object may be merged using the ffmpeg tool. Similarly, the audio clips belonging to the target object may also be merged using the ffmpeg tool.

[0087] ① The target video clip and the target audio clip are both relatively large. In order to reduce the use of computer memory and improve processing efficiency, in a specific implementation, the K-way merge sorting idea is introduced, and the formula is as follows:

[0088] merge_sort(p...r)=merge(mergesort(p...q),mergesort(q+1...r))

[0089] Among them, merge_sort(p...r) means sorting the array with subscripts from p to r; q represents the middle position between p and r, that is, (p+r) / 2.

[0090] Convert this sorting problem into two sub-problems mergesort(p...q) and mergesort(q+1...r). After the first and last word arrays are sorted, each forms an ordered small set of n elements:

[0091] s={x|x i ≤x j , i, j∈[0,n)}

[0092] When there are m small sets: s1, s2, ....s m

[0093] min=min(min(s1), min(s2),....min(s m ))

[0094] For example, suppose the target audio is segmented to obtain 9 audio files belonging to the target object, namely V1, V2, V3...V9.

[0095] Using 3-way merge sort, the 9 audio files are divided into three groups of nums1, num2, and nums3 in ascending order, as shown below:

[0096] nums1 = [1,2,8]

[0097] nums2 = [3,4,6]

[0098] nums3 = [5,7,9]

[0099] For ease of understanding, the three-way sequence is stored in the form of a matrix (as shown in Table 1 below). The value range of the horizontal coordinate x is 0, 1, 2, representing three columns, and the value range of the vertical coordinate is also 0, 1, 2, representing three rows. The first row (y = 0) is nums1, the second row (y = 1) is nums2, and the third row (y = 2) is nums3.

[0100] Table 1

[0101]

[0102]

[0103] (a) First, build a small top pile.

[0104] (ii) Add the minimum element of each path (i.e., the element in the first column) to the mini-heap. At this time, as shown in the above formula min=min(min(s_1),min(s_2),....min(s_m)), the top of the heap is the global minimum value in the k-path.

[0105] (3) Pop the top element of the heap and add the next element of the array where the top element is located to the heap.

[0106] (iv) Repeat steps (i) and (iii) until all data are read.

[0107] ② After K-way merge sorting, the previously independent audio clips (video clips) are merged into object audio (object video), and the merged object audio and object video are stored. The merged object audio and video are shown in Table 2:

[0108] Table 2

[0109] index Target object picture Clip 1 Audio and video files Start time End Time index Target object picture Clip 2 Audio and video files Start time End Time index Target object picture Clip 3 Audio and video files Start time End Time

[0110] In one embodiment, if the information of each target object has been obtained in advance, the feature extraction of the target object image in the storage record can be performed, and then the information of the current target object can be determined based on the extracted features and the time of the audio and video clips to determine the identity of the target object, forming a one-person-one-file data storage. The storage structure is shown in Table 3:

[0111] Table 3

[0112]

[0113] See also Figure 4 , Figure 44 is a schematic diagram of the structure of an embodiment of an audio and video segmentation device provided by the present application. The audio and video segmentation device 40 includes an acquisition module 41, a tracking module 42 and a segmentation module 43; the acquisition module 41 is used to acquire a target video and a target audio, wherein the target video and the target audio are acquired by collecting a target scene; the tracking module 42 is used to track the target object using the target video to obtain tracking data; the segmentation module 43 is used to segment the target audio and the target video from the target audio and the target video using the tracking data.

[0114] The tracking data includes the tracking end time. The tracking module 42 is used to determine the tracking end, including: in response to failure of tracking the target object, determining the tracking end of the target object.

[0115] Among them, the tracking module 42 is used to determine the failure of tracking of the target object, including: in response to the loss of tracking of the target object, obtaining at least two target video frames after the tracking is lost; for each target video frame, obtaining the similarity between the target video frame and any historical video frame before the tracking is lost; based on the similarities corresponding to each target video frame, determining the failure of tracking of the target object.

[0116] The tracking module 42 is used to obtain at least two target video frames after the tracking is lost, including: obtaining several video frames after the tracking is lost; and selecting a video frame whose quality meets the quality requirement from the several video frames as the target video frame.

[0117] Among them, the tracking module 42 is used to determine the failure of tracking the target object based on the similarity corresponding to each target video frame, including: for each target video frame, converting the similarity corresponding to the target video frame to obtain a result representation value; and comprehensively combining the result representation values ​​corresponding to each target video frame to determine the failure of tracking the target object.

[0118] Among them, the tracking module 42 is used to convert the similarity corresponding to the target video frame to obtain a result representation value, including: in response to the similarity corresponding to the target video frame being greater than or equal to the similarity threshold, the similarity corresponding to the target video frame is converted into a first result representation value; in response to the similarity corresponding to the target video frame being less than the similarity threshold, the similarity corresponding to the target video frame is converted into a second result representation value.

[0119] Among them, the first result characterization value and the second result characterization value are opposite in sign and equal in value, and the target video frames are an odd number; the tracking module 42 is used to integrate the result characterization values ​​corresponding to each target video frame to determine the failure of tracking the target object, including: summing the result characterization values ​​corresponding to each target video frame; in response to the sum being less than a preset value, determining that the tracking of the target object has failed.

[0120] Among them, the above-mentioned tracking data includes tracking moments, and the tracking moments include tracking start moments, tracking middle moments and tracking end moments. The tracking middle moments include moments when the tracking of the target object is lost and moments when the tracking of the target object continues; the segmentation module 43 is used to use the tracking data to segment the object audio and object video belonging to the target object from the target audio and target video, including: in response to obtaining any tracking moment other than the start moment, as the first target moment; using the target time interval, segmenting the target audio and target video to obtain target audio segments and target video segments; wherein the target time interval includes the second target moment, the first target moment and the moment between the first target moment and the second target moment, and the second target moment is the tracking moment before the first target moment; the audio and video segmentation device 40 also includes a merging module 44, and the merging module 44 is also used to merge the target audio segments belonging to the target object in response to the end of tracking of the target object to obtain the object audio of the corresponding target object; and merge the target video segments belonging to the target object to obtain the object video of the corresponding target object.

[0121] Wherein, the target video is acquired by using a video acquisition device, the target object is an object in the target scene whose distance from the video acquisition device meets a preset distance requirement; and / or, the target object includes a human face.

[0122] See also Figure 5 , Figure 5 : is a schematic diagram of the structure of an embodiment of an electronic device provided by the present application. The electronic device 50 includes a memory 51 and a processor 52 coupled to each other, and the processor 52 is used to execute program instructions stored in the memory 51 to implement the steps of any of the above-mentioned audio and video segmentation method embodiments. In a specific implementation scenario, the electronic device 50 may include but is not limited to: a microcomputer, a server, and in addition, the electronic device 50 may also include a mobile device such as a laptop computer and a tablet computer, which is not limited here.

[0123] Specifically, the processor 52 is used to control itself and the memory 51 to implement the steps of any of the above-mentioned audio and video segmentation method embodiments. The processor 52 can also be called a CPU (Central Processing Unit). The processor 52 may be an integrated circuit chip with signal processing capabilities. The processor 52 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 52 can be implemented by an integrated circuit chip.

[0124] See also Figure 6 , Figure 6 It is a structural diagram of an embodiment of a computer-readable storage medium provided by the present application. The computer-readable storage medium 60 of the embodiment of the present application stores a program instruction 61, and when the program instruction 61 is executed, it implements the method provided by any embodiment of the audio and video segmentation method of the present application and any non-conflicting combination. Among them, the program instruction 61 can form a program file and be stored in the above-mentioned computer-readable storage medium 60 in the form of a software product, so that a computer device (which can be a personal computer, a server, or a network device, etc.) executes all or part of the steps of the methods of each implementation method of the present application. The aforementioned computer-readable storage medium 60 includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a disk or an optical disk, or a terminal device such as a computer, a server, a mobile phone, and a tablet.

[0125] If the technical solution of this application involves personal information, the product using the technical solution of this application has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using the technical solution of this application has obtained the individual's separate consent before processing the sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, clear and prominent signs are set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that he or she agrees to the collection of his or her personal information; or on the device that processes personal information, the personal information processing rules are notified by obvious signs / information, and the individual's authorization is obtained through pop-up information or by asking the individual to upload his or her personal information; among them, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

[0126] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for segmenting audio and video, characterized in that: The method comprises: Acquire a target video and a target audio, wherein the target video and the target audio are acquired by capturing a target scene; Using the target video, tracking the target object to obtain tracking data; The tracking data is used to segment the target audio and the target video into the target audio and the target video.

2. The method according to claim 1, characterized in that The tracking data includes the tracking end time; the tracking end determination step includes: In response to failure of tracking the target object, it is determined that tracking of the target object is terminated.

3. The method according to claim 2, characterized in that The failure of tracking the target object includes: In response to the tracking loss of the target object, acquiring at least two target video frames after the tracking loss; For each of the target video frames, obtaining a similarity between the target video frame and any historical video frame before the tracking is lost; Based on the similarities corresponding to the target video frames, it is determined that tracking of the target object is invalid.

4. The method according to claim 3, characterized in that The obtaining of at least two target video frames after the tracking is lost includes: Acquire several video frames after the tracking is lost; A video frame whose quality meets the quality requirement is selected from the plurality of video frames as the target video frame.

5. The method according to claim 3, characterized in that: The determining, based on the similarities corresponding to the target video frames, that tracking of the target object is invalid comprises: For each of the target video frames, converting the similarity corresponding to the target video frame to obtain a result representation value; The result representation values ​​corresponding to the target video frames are combined to determine that tracking of the target object is invalid.

6. The method according to claim 5, characterized in that The converting the similarity corresponding to the target video frame to obtain a result representation value includes: In response to the similarity corresponding to the target video frame being greater than or equal to a similarity threshold, converting the similarity corresponding to the target video frame into a first result representation value; In response to the similarity corresponding to the target video frame being less than the similarity threshold, the similarity corresponding to the target video frame is converted into a second result representation value.

7. The method according to claim 6, characterized in that The first result characterizing value and the second result characterizing value are opposite in sign and equal in value, and the number of the target video frames is an odd number; The step of synthesizing the result representation values ​​corresponding to the target video frames to determine that tracking of the target object is invalid includes: Sum the result representation values ​​corresponding to each of the target video frames; In response to the sum being less than a preset value, it is determined that tracking of the target object is invalid.

8. The method according to claim 7, characterized in that The method further comprises: In response to the sum being greater than the preset value, it is determined that the tracking of the target object continues.

9. The method according to claim 1, characterized in that: The tracking data includes tracking moments, wherein the tracking moments include tracking start moments, tracking middle moments, and tracking end moments, wherein the tracking middle moments include moments when the tracking of the target object is lost and moments when the tracking of the target object is continued; the step of using the tracking data to segment the object audio and the object video belonging to the target object from the target audio and the target video includes: In response to obtaining any tracking time other than the start time, use it as the first target time; Using a target time interval, segmenting the target audio and target video to obtain a target audio segment and a target video segment; wherein the target time interval includes a second target moment, the first target moment, and a moment between the first target moment and the second target moment, and the second target moment is a tracking moment before the first target moment; The method further comprises: In response to the completion of tracking of the target object, merging target audio segments belonging to the target object to obtain object audio corresponding to the target object; and The target video segments belonging to the target object are merged to obtain an object video corresponding to the target object.

10. The method according to claim 1, characterized in that The target video is collected by a video collection device, and the target object is an object in the target scene whose distance from the video collection device meets a preset distance requirement; and / or, The target object includes a human face.

11. An audio and video segmentation device, characterized in that: The device comprises: An acquisition module, used to acquire a target video and a target audio, wherein the target video and the target audio are acquired by collecting a target scene; A tracking module, used to track the target object using the target video to obtain tracking data; The segmentation module is used to segment the target audio and the target video from the target audio and the target video by using the tracking data.

12. An electronic device, characterized in that: The electronic device comprises a memory and a processor, wherein the memory stores program instructions, and the program instructions can be executed to implement the method according to any one of claims 1-10.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store program instructions, and the program instructions can be executed to implement the method according to any one of claims 1-10.

Images