Video processing method, device and electronic system

Abstract

The present invention provides a video processing method, apparatus and electronic system. Among them, the method includes: obtaining a video stream for target tracking; obtaining state parameters corresponding to at least one target trajectory in the video stream; where the state parameters are used to characterize the degree of proximity of the target trajectory to other trajectories in the video stream; selecting one or more target detection frames according to the state parameters of the target trajectory. In this video processing method, by considering the state parameters corresponding to the target trajectory in the video stream, the consideration factors for the target detection frame are enriched, thereby improving the accuracy of the target detection frame (i.e., the push map result), and having good practical value.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a video processing method, device and electronic system. Background Art

[0002] Image push logic is widely used in security, access, and other product implementations. Finding appropriate, quality-compliant target objects (e.g., faces and bodies) from a specified tracking track is crucial for various related products.

[0003] Common push logic includes optimal push, fastest push, and interval push. Optimal push is to push images at the end of a track and push the best quality image in that track. Fastest push is to push images as soon as the quality of a certain frame meets the required level for a track. Interval push is to push images at specified time intervals during the lifetime of a track.

[0004] However, for dense scenes, ID switches are prone to occur under the above-mentioned image pushing logic. For example, for a certain track, it starts tracking target object A. At a certain moment, the tracked target object changes to B. At this time, the track undergoes an ID switch. Afterwards, the track may track target object A again, or it may track target object C. In other words, the track undergoes another ID switch. After the track undergoes an ID switch, the detection box list corresponding to the track may contain different target objects, such as target object A, target object B, and target object C mentioned above.

[0005] For scenarios where ID switching is prone to occur, how to push the tracking trajectory to effectively ensure the pushing quality has not yet been proposed. Summary of the Invention

[0006] In view of this, the purpose of the present invention is to provide a video processing method, device and electronic system, which improves the rationality of trajectory mapping by considering the state parameters corresponding to the target trajectory in the video stream, thereby effectively ensuring the mapping quality.

[0007] In a first aspect, an embodiment of the present invention provides a video processing method, which includes: obtaining a video stream for target tracking; obtaining state parameters corresponding to at least one target track in the video stream; wherein the state parameters are used to characterize the degree of proximity between the target track and other tracks in the video stream; and selecting one or more target detection frames based on the state parameters of the target track.

[0008] In combination with the first aspect, an embodiment of the present invention provides a first possible implementation manner of the first aspect. Among them, the step of selecting one or more target detection frames according to the state parameters of the target trajectory includes: determining the segment to which the current video frame of the video stream belongs in the target trajectory according to the state parameters of the target trajectory; wherein, the congestion states corresponding to adjacent segments of the target trajectory are different; selecting one or more target detection frames corresponding to the target trajectory according to the congestion states corresponding to each segment of the target trajectory.

[0009] In combination with the first possible implementation manner of the first aspect, an embodiment of the present invention provides a second possible implementation manner of the first aspect. Among them, the step of selecting one or more target detection frames corresponding to the target trajectory according to the congestion states corresponding to each segment of the target trajectory includes: performing smoothing processing on the congestion states corresponding to each segment of the target trajectory; selecting one or more target detection frames corresponding to the target trajectory according to the smoothed congestion states corresponding to each segment of the target trajectory.

[0010] In combination with the first possible implementation manner or the second possible implementation manner of the first aspect, an embodiment of the present invention provides a third possible implementation manner of the first aspect. Among them, the step of selecting one or more target detection frames corresponding to the target trajectory according to the congestion states corresponding to each segment of the target trajectory includes: obtaining candidate detection frames corresponding to each segment of the target trajectory for the target object; determining one or more target detection frames of the target trajectory according to a preset image quality weight and / or a congestion state weight corresponding to each segment.

[0011] In combination with the third possible implementation manner of the first aspect, an embodiment of the present invention provides a fourth possible implementation manner of the first aspect. Among them, the above target object includes a local object of a specified target and an overall object of the specified target; the step of selecting one or more target detection frames corresponding to the target trajectory according to the congestion states corresponding to each segment of the target trajectory includes: determining a first target detection frame corresponding to the local object and a second target detection frame corresponding to the overall object according to the congestion states corresponding to each segment of the target trajectory; wherein, the first target detection frame and the second target detection frame belong to the same video frame or different video frames; using the first target detection frame and the second target detection frame as the target detection frames of the specified target.

[0012] In combination with the fourth possible implementation manner of the first aspect, an embodiment of the present invention provides a fifth possible implementation manner of the first aspect. Among them, the step of determining a first target detection frame corresponding to the local object and a second target detection frame corresponding to the overall object according to the congestion states corresponding to each segment of the target trajectory includes: determining non-congested segments according to the congestion states corresponding to each segment of the target trajectory; determining the first target detection frame corresponding to the local object and the second target detection frame corresponding to the overall object from the non-congested segments.

[0013] Combined with the fourth possible implementation manner of the first aspect, the embodiment of the present invention provides a sixth possible implementation manner of the first aspect, wherein the above-mentioned crowded state corresponds to a crowding level, and the crowding level corresponds to the degree of crowding from crowded to sparse from high to low; the step of determining the first target detection box corresponding to the local object and the second target detection box corresponding to the overall object according to the crowded states corresponding to the respective segments of the target trajectory includes: for the first segment of the target trajectory with a crowding level greater than the level threshold, selecting candidate detection boxes corresponding to the local object and the overall object respectively in each same video frame corresponding to the first segment; for the second segment of the target trajectory with a crowding level not greater than the level threshold, selecting candidate detection boxes corresponding to the local object and the overall object respectively in different video frames corresponding to the second segment; determining the first target detection box corresponding to the local object and the second target detection box corresponding to the overall object according to the preset image quality weight and the crowded state weight corresponding to each segment.

[0014] Combined with any one of the fourth to sixth possible implementation manners of the first aspect, the embodiment of the present invention provides a seventh possible implementation manner of the first aspect, wherein the above-mentioned specified target is a person, the local object is a face, and the overall object is a human body; or, the above-mentioned specified target is a vehicle, the local object is a license plate, and the overall object is a vehicle body.

[0015] Combined with the first aspect, the embodiment of the present invention provides an eighth possible implementation manner of the first aspect, wherein the step of obtaining the state parameter corresponding to at least one target trajectory in the video stream includes: sequentially taking each video frame in the video stream as the current video frame according to the time sequence of the video frames in the video stream, and performing the following operations: obtaining the representative information of the target trajectory; wherein, the representative information is used to characterize the feature information and / or position information of the target object corresponding to the target trajectory; the position information includes the prediction box of the target trajectory or the coordinate information of the detection box corresponding to the target object in the previous frame of the current video frame; wherein, the prediction box is the position box of the target object corresponding to the target trajectory predicted in the current video frame according to the positions of the detection boxes in the existing video frames before the current video frame of the target trajectory; determining the state parameter corresponding to the target trajectory for the current video frame according to the representative information and the detection box corresponding to the target object in the current video frame.

[0016] Combined with the eighth possible implementation manner of the first aspect, the embodiment of the present invention provides a ninth possible implementation manner of the first aspect. Among them, the state parameters of the above-mentioned target trajectory include a plurality of values corresponding one by one to video frames; the step of determining the state parameter of the target trajectory corresponding to the current video frame according to the representative information and the detection box corresponding to the target object in the current video frame includes: obtaining the target parameters corresponding to the detection boxes of each target object in the representative information and the current video frame; where the target parameters include: IoU and / or feature similarity; determining the state parameter of the target trajectory corresponding to the current video frame according to the representative information and the target parameters corresponding to each detection box.

[0017] Combined with the ninth possible implementation manner of the first aspect, the embodiment of the present invention provides a tenth possible implementation manner of the first aspect. Among them, the step of determining the state parameter of the target trajectory corresponding to the current video frame according to the representative information and the target parameters corresponding to each detection box includes: counting the number of target parameters greater than the first threshold in the representative information and the target parameters corresponding to each detection box to obtain a first value, and determining the state parameter of the target trajectory corresponding to the current video frame according to the first value; or calculating the sum of the representative information and the target parameters corresponding to each detection box to obtain a second value, and determining the state parameter of the target trajectory corresponding to the current video frame according to the second value.

[0018] Combined with the ninth possible implementation manner of the first aspect, the embodiment of the present invention provides an eleventh possible implementation manner of the first aspect. Among them, the step of determining the state parameter of the target trajectory corresponding to the current video frame according to the representative information and the target parameters corresponding to each detection box includes: determining the loss values of the representative information and each detection box according to the representative information and the target parameters corresponding to each detection box; where the loss value is negatively correlated with the target parameter of the detection box; counting the number of loss values of the representative information and each detection box less than the second threshold to obtain a third value, and determining the state parameter of the target trajectory corresponding to the current video frame according to the third value.

[0019] Combined with the ninth possible implementation manner of the first aspect, the embodiment of the present invention provides the twelfth possible implementation manner of the first aspect. Among them, the step of determining the state parameter of the target trajectory corresponding to the current video frame according to the representative information and the target parameters corresponding to each detection box includes: determining a loss matrix according to the representative information of each trajectory and the target parameters corresponding to each detection box in the current video frame; where the elements in the loss matrix are the loss values corresponding to the trajectory and the detection box, and the loss value is negatively correlated with the target parameter corresponding to the detection box; based on the loss matrix, allocate trajectories to each detection box in the current video frame, and perform the following operations on the detection boxes with allocated trajectories: find the first loss value of the first detection box corresponding to the allocated first trajectory and the second loss value of the first detection box corresponding to other trajectories except the first trajectory from the loss matrix; calculate the difference between the first loss value and the second loss value, if the absolute value of the difference is less than the third threshold, add 1 to the statistical values of the first trajectory and other trajectories corresponding to the current video frame respectively; according to the statistical value of the target trajectory corresponding to the current video frame at the end of the operation, determine the state parameter of the target trajectory corresponding to the current video frame.

[0020] Combined with the first possible implementation manner of the first aspect, the embodiment of the present invention provides the thirteenth possible implementation manner of the first aspect. Among them, the step of determining the segment to which the current video frame of the video stream belongs in the target trajectory according to the state parameter of the target trajectory includes: segmenting the current video frame corresponding to the target trajectory according to the state parameter of the target trajectory; where the segmentation includes at least one crowded segment and / or at least one non-crowded segment.

[0021] Combined with the thirteenth possible implementation manner of the first aspect, the embodiment of the present invention provides the fourteenth possible implementation manner of the first aspect. Among them, the step of segmenting the current video frame corresponding to the target trajectory according to the state parameter of the target trajectory includes one of the following: determining the segment to which the current video frame corresponding to the target trajectory belongs according to the current state parameter of the target trajectory; in the existing video frames corresponding to the target trajectory, calculate the reference value of the state parameters of a specified number of consecutive video frames starting from the current video frame, and determine the segment to which the current video frame belongs according to the reference value; where the reference value includes a sum value or an average value; update the cumulative statistic according to the current state parameter of the target trajectory, and determine the segment to which the current video frame corresponding to the target trajectory belongs according to the updated cumulative statistic; where the cumulative statistic is used to characterize the crowded state of the most recent segment of the target trajectory.

[0022] Second aspect, an embodiment of the present invention further provides a video processing device, which includes: a video stream acquisition module for acquiring a video stream for target tracking; a state parameter acquisition module for acquiring state parameters corresponding to at least one target trajectory in the video stream; wherein the state parameters are used to characterize the proximity of the target trajectory to other trajectories in the video stream; a detection box selection module for selecting one or more target detection boxes according to the state parameters of the target trajectory.

[0023] Third aspect, an embodiment of the present invention further provides an electronic system, which includes: an image acquisition device, a processing device, and a storage device; wherein the image acquisition device is used to acquire a video stream; a computer program is stored on the storage device, and when the computer program is run by the processing device, it executes the video processing method of the first aspect.

[0024] Fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is run by the processing device, it executes the steps of the video processing method of the first aspect above.

[0025] The embodiments of the present invention bring the following beneficial effects:

[0026] The embodiments of the present invention provide a video processing method, device, and electronic system. By acquiring the state parameters corresponding to the target trajectory in the video stream to select one or more target detection boxes (i.e., the map pushing result), this video processing method that considers the state variables of the proximity of the target trajectory to other trajectories in the video stream can make the factors considered in map pushing more comprehensive, thereby improving the rationality of map pushing, effectively ensuring the quality of trajectory map pushing, and having good practical value.

[0027] Other features and advantages of the present disclosure will be described in the subsequent specification, or some features and advantages can be inferred from the specification or determined without doubt, or can be known by implementing the above technologies of the present disclosure.

[0028] To make the above objects, features, and advantages of the present disclosure more obvious and understandable, the following specific preferred embodiments are given below and are described in detail in conjunction with the accompanying drawings as follows. Description of the Drawings

[0029] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required for the description of the specific embodiments or the prior art. Obviously, the following drawings are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0030] Figure 1Schematic diagram of a structure of an electronic system provided by an embodiment of the present invention;

[0031] Figure 2 Flowchart of a video processing method provided by an embodiment of the present invention;

[0032] Figure 3 Flowchart of another video processing method provided by an embodiment of the present invention;

[0033] Figure 4 Flowchart of another video processing method provided by an embodiment of the present invention;

[0034] Figure 5 Flowchart of a method for obtaining another state parameter provided by an embodiment of the present invention;

[0035] Figure 6 Schematic diagram of the principle for calculating IoU provided by an embodiment of the present invention;

[0036] Figure 7 Flowchart of another video processing method provided by an embodiment of the present invention;

[0037] Figure 8 Schematic diagram of a video processing device provided by an embodiment of the present invention;

[0038] Figure 9 Schematic diagram of a structure of an electronic device provided by an embodiment of the present invention. Detailed implementation manners

[0039] To make the objectives, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Apparently, the described embodiments are some but not all of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0040] Aiming at the problem of unsatisfactory map inference results in the existing method for inferring maps based on tracking trajectories, the embodiments of the present invention provide a video processing method, device and electronic system. By considering the congestion states corresponding to each segment of the target trajectory, the factors considered in map inference can be made more comprehensive, thereby improving the rationality of trajectory map inference, effectively ensuring the quality of trajectory map inference, and having good practical value.

[0041] For the convenience of understanding this embodiment, first, refer to Figure 1 the schematic diagram of the structure of the electronic system shown in. This electronic system can be used to implement the video processing method, device and electronic system of the embodiments of the present invention.

[0042] As shown inFigure 1 Schematic structural diagram of an electronic system shown. The electronic system 100 includes one or more processing devices 102, one or more storage devices 104, an input device 106, an output device 108, and one or more image acquisition devices 110. These components are interconnected through a bus system 112 and / or other forms of connection mechanisms (not shown). It should be noted that Figure 1 The components and structure of the electronic system 100 shown are exemplary rather than restrictive. According to needs, the electronic system may also have other components and structures.

[0043] The processing device 102 can be a server, a smart terminal, or a device or apparatus including a central processing unit (CPU) or other forms of processing units with data processing capabilities and / or instruction execution capabilities. It can process the data of other components in the electronic system 100 and can also control other components in the electronic system 100 to perform video processing functions.

[0044] The storage device 104 can include one or more computer program products. The computer program products can include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory can include, for example, random access memory (RAM) and / or cache memory, etc. Non-volatile memory can include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions can be stored on the computer-readable storage medium. The processing device 102 can run the program instructions to implement the client functions in the embodiments of the present invention below (implemented by the processing device) and / or other desired functions. Various application programs and various data can also be stored in the computer-readable storage medium, such as various data used and / or generated by the application programs, etc.

[0045] The above input device 106 can be a device used by a user to input instructions and can include one or more of a keyboard, a mouse, a microphone, a touch screen, etc., or can also be a data input interface.

[0046] The output device 108 can output various information (such as images or sounds) to the outside (for example, to the user) and can include one or more of a display, a speaker, etc., or can also be a data output interface.

[0047] The image acquisition device 110 can acquire images, including still images, video streams, etc., such as the user's face image, etc., and store the acquired images in the storage device 104 for use by other components. The image acquisition device 110 can be a separate camera, a camera in a mobile terminal, or an image sensor in a snapshot camera. It should be understood that the image acquisition device 110 is only an example, and the electronic system 100 may not include the image acquisition device 110. In this case, other devices with video stream acquisition capabilities can be used to acquire images and send the acquired images to the electronic system 100.

[0048] Exemplarily, the devices for implementing the video processing method, apparatus, and electronic system according to the embodiments of the present invention can be integrally arranged or dispersedly arranged. For example, the processing device 102, the storage device 104, the input device 106, and the output device 108 can be integrally arranged, while the image acquisition device 110 is arranged at a specified position where a video stream can be acquired. When the devices in the above electronic system are integrally arranged, the electronic system can be implemented as an intelligent terminal such as a camera, a smart phone, a tablet computer, a computer, a vehicle-mounted terminal, etc.

[0049] Embodiments of the present invention provide a video processing method, which is applied to an electronic device configured with the above electronic system. As Figure 2 shown, the method includes the following steps:

[0050] Step S202, obtaining a video stream for target tracking;

[0051] Among them, the above video stream can be an original video stream acquired by an image acquisition device, or a video stream after preprocessing the original video stream. Here, the preprocessing includes but is not limited to scaling processing and noise reduction processing of the video stream, etc., and can be specifically set according to actual application situations.

[0052] Step S204, obtaining state parameters corresponding to at least one target trajectory in the video stream; among them, the state parameters are used to characterize the proximity of the target trajectory to other trajectories in the video stream.

[0053] In practical applications, the video stream includes multiple video frames, and each video frame contains multiple target objects. For example, when performing target tracking on a person, the target objects include a face and / or a human body; or when performing target tracking on a vehicle, the target objects include a vehicle and / or a license plate.

[0054] In order to track the target object, trajectories will be assigned to the target object according to the position of the target object in the current adaptation frame and / or the characteristics of the target object, as well as the target characteristics corresponding to the existing trajectories and / or the position of the detection frame corresponding to the trajectory. For example, target object 1 is assigned to Track1, target object 2 is assigned to Track2, etc.

[0055] For a newly added target object in the current video frame of the video stream (i.e., the target object does not exist in the video frames before the current video frame of the video stream), a new track can be assigned to the newly added target object. Assume that the existing tracks are Track1 - Track5, and the current video frame contains 6 target objects. Among them, 5 target objects are respectively assigned to Track1 - Track5, and the 6th target object does not match a suitable track. Then a new track Track6 is generated, and the 6th target object is assigned to Track6, so that there will be a correspondence between the target objects and the tracks.

[0056] Since the scenes corresponding to each video frame may be different, the positional relationships between the tracks corresponding to different target objects and other tracks will also change. For example, in a scene with a large flow of people, the tracks corresponding to each target object (such as a human body) may be relatively close, and even partial overlaps may occur; while in a scene with a small flow of people, the tracks corresponding to each target object may be relatively far apart from each other. To characterize the magnitude of the flow of people where a certain target track in the video stream is located, this embodiment introduces a state parameter corresponding to the target track, and this state parameter is used to characterize the degree of proximity between the target track and other tracks in the video stream.

[0057] It should be noted that the above-mentioned target track is merely a name for the track corresponding to a target object, and each target object will correspond to a target track according to the track assignment situation.

[0058] Step S206, select one or more target detection frames according to the state parameter of the target track.

[0059] Specifically, according to the state parameter of the target track, determine the segment to which the current video frame of the video stream belongs in the target track; among them, the congestion states corresponding to adjacent segments of the target track are different; and according to the congestion states corresponding to each segment of the target track, select one or more target detection frames corresponding to the target track. In practical applications, since each target track has a state parameter corresponding to the current video frame, and the state parameter is used to characterize the degree of proximity between the target track and other tracks in the current video frame, it can be determined based on the state parameter whether there are interference items around the target object corresponding to the target track. For example, in a dense or congested scene, the state parameter is relatively large, indicating that there are more interference items around the target object in the current video frame, and these interference items will affect the map inference of the target track. Therefore, through this state parameter, the segment to which the current video frame in the video stream belongs in the target track can be determined.

[0060] Among them, the segmentation of the target trajectory is to group the video frames with the same congestion state according to the time sequence of the video frames in the video stream, and the congestion states of adjacent segments are different. For example, for the target trajectory 1, the video frames with the same or similar state parameters in the video stream are divided into the same segment according to the time sequence, and the congestion states corresponding to adjacent segments are different. It should be noted that since the state parameters of each target trajectory in the current video frame may not be the same, for the current video frame, it may belong to the first congested segment in the target trajectory 1 and at the same time belong to the second congested segment or non-congested segment in the target trajectory 2, where the first congested segment and the second congested segment are segments corresponding to different congestion states respectively.

[0061] In addition, considering that the map pushing strategy (such as optimal, fastest or interval map pushing) can be selected according to the actual application scenario, when the target trajectory reaches the time when map pushing is required, the target detection box selected in the above step S206 can be output. In this embodiment, based on the fact that the target trajectory has been segmented, the congestion state of each segment can be considered, and the quality of most of the images of the target object in this segment can be inferred from the congestion state of this segment. Therefore, map pushing of the target trajectory can be performed based on the congestion states of each segment, improving the accuracy of the map pushing result.

[0062] The video processing method provided by the embodiment of the present invention selects one or more target detection boxes, that is, the map pushing result, by obtaining the state parameters corresponding to the target trajectory in the video stream. This video processing method that considers the state variables of the proximity between the target trajectory and other trajectories in the video stream during map pushing can make the factors considered in map pushing more comprehensive, thereby improving the rationality of the map pushing result, effectively ensuring the quality of trajectory map pushing, and having good practical value.

[0063] Optionally, in practical applications, since the target object is moving, there may be a situation where the segments of the target trajectory are highly jumpy. For example, video frames 1-6 are segments with a congestion state of 1, video frame 7 is a segment with a congestion state of 3, and video frames 8-16 are all segments with a congestion state of 1, and the state parameter of a certain video frame is quite different from the state parameters of the video frames in the adjacent segment. At this time, in order to reduce this jumpiness, the method further includes: smoothing the congestion states corresponding to each segment of the target trajectory; selecting one or more target detection boxes corresponding to the target trajectory according to the smoothed congestion states corresponding to each segment of the target trajectory.

[0064] Among them, the above-mentioned smoothing process can be similar to processes such as averaging the congestion state. Continuing with the previous example, after the above-mentioned video frame 7 is smoothed, the congestion state can be 2 or 1. By smoothing the congestion states corresponding to the segments in the target trajectory, the accuracy of the state of each segment can be ensured, and the over-segmentation of the target trajectory can be avoided. While ensuring the accuracy of the target detection box (i.e., the result of pushing the image), the computational complexity is also reduced.

[0065] In practical applications, according to the state parameters of the target trajectory, the current video frame corresponding to the target trajectory can be segmented; among them, the segmentation includes at least one congested segment and / or at least one non-congested segment. One possible way to determine the segmentation includes the following process: Determine the congestion level corresponding to the state parameters of the target trajectory in the current video frame, and use the segment corresponding to this congestion level as the segment to which the current video frame belongs in the above-mentioned target trajectory. Specifically, the mapping relationship between the state parameter range and the congestion level can be pre-stored in the electronic device. For example, for state parameters 1.2 to 1.5, the corresponding congestion level is the first-level congestion, and for state parameters 1.5 to 1.8, the corresponding congestion level is the second-level congestion, etc. Thus, according to the state parameters of each target trajectory in the current video frame, the corresponding congestion level is determined, and the segment corresponding to this congestion level is used as the segment of the current video frame. This way of determining the segmentation is relatively simple and fast, facilitating the quick determination of the segment where the current video frame is located for each target trajectory.

[0066] In addition, another possible way to determine the segmentation includes the following process: In the existing video frames corresponding to the target trajectory (i.e., the video frames received by the electronic device before the current video frame in the video stream), calculate the reference value of the state parameters of a specified number of consecutive video frames starting from the current video frame, and determine the segment to which the current video frame belongs according to this reference value. For example, use the segment corresponding to the congestion level of this reference value as the segment to which the current video frame belongs in the above-mentioned target trajectory.

[0067] Among them, the above reference values include but are not limited to the sum value or the average value, and can be specifically set according to the actual situation. Here, the sum value is taken as an example for illustration. Specifically, for the target trajectory 1 in the current video frame, state parameters of the target trajectory 1 such as 1.11 are calculated. For the existing video frames corresponding to the target trajectory 1, for a consecutive specified number such as 3 video frames before the current video frame, the state parameters of the target trajectory in 2 consecutive video frames before the current video frame are respectively calculated, such as 1.23 and 1.21, and the sum of the state parameters of the current video frame and the previous 2 video frames (1.11 + 1.23 + 1.21) is calculated to obtain a sum value such as 3.55. At this time, the electronic device determines the segment to which the current video frame belongs according to the mapping relationship between the pre-stored state parameter range and the congestion level; or a sum value threshold can also be set to determine whether the sum value is greater than the sum value threshold, and the segment to which the current video frame belongs is determined according to the judgment result.

[0068] Alternatively, in addition to calculating the sum value of the state parameters of the consecutive specified number of video frames starting from the current video frame as described above, a state parameter threshold can also be set, and the number of state parameters greater than the state parameter threshold among the state parameters of the current video frame and the specified number of video frames is determined, and the segment to which the current video frame belongs is determined according to the number. Specifically, it can be set according to the actual situation, and the embodiments of the present invention do not limit this.

[0069] For ease of understanding, an example is given here. For example, for the target trajectory 1, for 10 consecutive video frames from time t0 to time t9, the state parameters corresponding to the target trajectory 1 are shown in Table 1 below:

[0070] Table 1

[0071] Moment t0 t1 t2 t3 t4 t5 t6 t7 t8 t9 State parameter 1.11 1.21 1.13 1.23 1.33 1.45 1.61 2.71 2.1 2.11

[0072] As shown in Table 1, since there is no video frame before t0, the state parameter of the target trajectory 1 in the video frame at time t0 is 1.11. Assuming that the specified number is 3 and the sum value threshold is 4, for the first segment corresponding to t0 - t2, the sum of the state parameters is 1.11 + 1.21 + 1.13 = 3.45. This sum value is less than the sum value threshold 4, and the first segment is marked as a sparse state.

[0073] At this time, if the current video frame is the video frame at time t3, then according to the sum of the state parameters of t1 - t3, 1.21 + 1.13 + 1.23 = 3.57, this sum value is still less than the sum value threshold 4, so the segment to which the video frame at time t3 belongs is the first segment; similarly, if the current video frame is the video frame at time t4, then according to the sum of the state parameters of t2 - t4, 1.13 + 1.23 + 1.33 = 3.69, this sum value is still less than the sum value threshold 4, so the segment to which the video frame at time t4 belongs is also the first segment.

[0074] If the current video frame is the video frame at time t5, then according to the sum of the state parameters from t3 to t5, which is 1.23 + 1.33 + 1.45 = 4.01, and this sum value is greater than the sum value threshold of 4. At this time, the segment to which the video frame at time t5 belongs is the second segment, such as the crowded state. Similarly, it can be determined that the video frames at times t6 - t9 all belong to the second segment.

[0075] In addition, the state parameter threshold can also be set to 1.4. Since the number of state parameters from t0 to t4 that are greater than the state parameter threshold of 1.4 is 0, the segments to which the video frames from t0 to t4 belong are the first segment, that is, the sparse state. Starting from time t5, the state parameters are all greater than the state parameter threshold of 1.4. The segments to which the video frames from t5 to t9 belong can be the second segment, that is, the crowded state. The number of state parameters greater than the state parameter threshold can also be set. For example, if the number is 4, it can be determined that the segments to which the video frames from t5 to t8 belong are the second segment, that is, the first crowded state, and the segments to which the video frames at time t9 belong are the third segment, that is, the second crowded state, etc. Thus, the video frames are divided according to the state parameters of the target trajectory and are in different segments.

[0076] In addition to the above segmentation methods, the segmentation of the video frame can also be determined according to the following method, which specifically includes: updating the cumulative statistic according to the current state parameter of the target trajectory, and determining the segment to which the current video frame corresponding to the target trajectory belongs according to the updated cumulative statistic; where the cumulative statistic is used to characterize the crowded state of the nearest segment of the target trajectory.

[0077] Specifically, if the current state parameter of the target trajectory is greater than the fourth threshold, then add 1 to the cumulative statistic to obtain the updated cumulative statistic, and determine the segment to which the current video frame corresponding to the target trajectory belongs according to the updated cumulative statistic. For example, use the segment of the crowded state level corresponding to the updated cumulative statistic as the segment to which the current video frame belongs; conversely, if the state parameter of the target trajectory is not greater than the fourth threshold and the number of times the cumulative statistic has not been updated reaches the fifth threshold, reset the cumulative statistic to the preset initial value, and use the segment of the crowded state level corresponding to the preset initial value as the segment to which the current video frame belongs; if the state parameter of the current video frame is not greater than the fourth threshold and the number of times the cumulative statistic has not been updated does not reach the fifth threshold, use the segment of the crowded state level corresponding to the current cumulative statistic as the segment to which the current video frame belongs.

[0078] For ease of understanding, an example is given here. Suppose the fourth threshold is 2. If the state parameter of the target trajectory is 2.23 currently, that is, the state parameter of the current video frame is greater than the fourth threshold, at this time, the cumulative statistic is incremented by 1, and the updated cumulative statistic is obtained as 2. Then, the segment corresponding to the updated cumulative statistic in the crowding state level is used as the segment to which the current video frame belongs. For example, the cumulative statistic 2 corresponds to the first crowding state, the cumulative statistic 3 corresponds to the second crowding state, the cumulative statistic 4 corresponds to the third crowding state, etc. Therefore, at this time, the segment to which the current video frame belongs is the segment in the first crowding state. If the state parameter of the current video frame is 1.23, that is, the state parameter of the current video frame is not greater than the fourth threshold, then the cumulative statistic is not updated for the first time. At this time, if the number of times the cumulative statistic is not updated does not reach the fifth threshold, such as 3 times, the segment corresponding to the current cumulative statistic in the crowding state level is used as the segment to which the current video frame belongs. For example, the current cumulative statistic is 3, that is, the cumulative statistic is 3 at this time and it is not updated for the first time. Then, at this time, the segment to which the current video frame belongs is the segment in the second crowding state. Similarly, if the state parameter of the next video frame is still not greater than the fourth threshold, then the cumulative statistic is not updated for the second time and still does not reach the fifth threshold. Then, the segment to which the next video frame of the target trajectory belongs is still the segment in the second crowding state. For the next video frame, if its state parameter is still not greater than the fourth threshold, then the cumulative statistic is not updated for the third time and reaches the fifth threshold. At this time, the cumulative statistic needs to be reset to the preset initial value, such as 0, and the segment corresponding to the preset initial value in the crowding state level is used as the segment to which the current video frame belongs. It should be noted that the above fourth threshold, fifth threshold, preset initial value, etc. can be set according to the actual situation, and the embodiments of the present invention do not limit this.

[0079] According to the crowding states of each segment of the above target trajectory, alternative detection frames for each segment can be obtained, and thus the target detection frame of the target trajectory, that is, the map pushing result, can be realized based on multiple alternative detection frames. Optionally, for the optimal map pushing logic, the target alternative detection frame with the best image quality can be determined from the alternative detection frames, and this target alternative detection frame can be used as the final target detection frame of the corresponding target trajectory; for the fastest map pushing logic, according to the time sequence of the target trajectory, if the alternative detection frame of the current segment meets the quality requirements, the alternative detection frame of the current segment can be used as the final target detection frame of the target trajectory; and for the interval map pushing logic, map pushing can be performed according to the time sequence and a preset interval, such as every two segments, and one or more target detection frames of the target trajectory can be determined based on the alternative detection frames that meet the quality requirements.

[0080] Based on the Figure 2 method shown above, the embodiments of the present invention also provide another video processing method, which focuses on describing the process of selecting one or more target detection frames corresponding to the target trajectory according to the crowding states corresponding to each segment of the target trajectory; such asFigure 3 As shown, the method includes the following steps:

[0081] Step S302, obtaining a video stream for target tracking;

[0082] Step S304, obtaining state parameters corresponding to at least one target trajectory in the video stream;

[0083] Among them, the state parameters are used to characterize the degree of proximity of the target trajectory to other trajectories in the video stream.

[0084] Step S306, determining the segment to which the current video frame of the video stream belongs in the target trajectory according to the state parameters of the target trajectory;

[0085] Among them, the congestion states corresponding to adjacent segments of the target trajectory are different.

[0086] The above steps S302 to S306 can refer to the foregoing embodiments, and the embodiments of the present invention will not be elaborated herein.

[0087] Step S308, obtaining candidate detection frames corresponding to each segment of the target trajectory for the target object;

[0088] Step S310, determining one or more target detection frames of the target trajectory according to a preset image quality weight and / or a congestion state weight corresponding to each segment.

[0089] Specifically, the weights of each segment in the electronic device include an image quality weight and / or a congestion state weight, so that the candidate detection frames of each segment can be calculated according to the weights to obtain a comprehensive score for each candidate detection frame. In practical applications, the image quality weight and the congestion state weight in the above electronic device can be fixed values or can be set according to actual situations, and the embodiments of the present invention do not limit this.

[0090] One possible way to determine the target detection box includes: calculating only based on the image quality weights of each candidate detection box corresponding to each segment of the target trajectory for the target object, obtaining the image quality scores of each candidate detection box, and determining one or more candidate detection boxes with image quality scores higher than the preset image quality score threshold as the target detection box. For example, for some sparse scenarios, the influence of the congestion state of each segment can be ignored at this time, and only the image quality scores of each candidate detection box are calculated, and one or more candidate detection boxes with image quality scores higher than 90 points are determined as the target detection box corresponding to the target trajectory. Another possible way to determine the target detection box includes: calculating the scores of each candidate detection box corresponding to each segment of the target trajectory only based on the congestion weights corresponding to each segment, and determining one or more candidate detection boxes with scores higher than the preset score threshold as the target detection box. For example, for some congested scenarios, the image quality of the target object in each candidate detection box meets the quality requirements. At this time, only the scores of each candidate detection box are calculated based on the congestion state weights, and one or more candidate detection boxes with scores higher than the preset score threshold, such as 85 points, are determined as the target detection box corresponding to the target trajectory.

[0091] In addition, in practical applications, in order to ensure the accuracy of the target object in the target detection box, it is often necessary to set both the image quality weight of the candidate detection box and the congestion state weight corresponding to each segment at the same time, calculate the comprehensive score of the candidate detection box according to the image quality weight of each candidate detection box and the congestion state weight corresponding to the segment where it is located, compare the comprehensive score with the preset comprehensive score threshold, and determine one or more candidate detection boxes higher than the comprehensive score threshold as the target detection box, thus ensuring the image quality of the target detection box in the congested state and improving the accuracy of the target detection box.

[0092] The above video processing method calculates the comprehensive scores of the candidate detection boxes corresponding to each segment and determines the final one or more target detection boxes of the target trajectory, that is, the map pushing result. Compared with the existing method of directly pushing the map for the entire target trajectory, it reduces the problem of incorrect binding caused by ID Switch in Track, resulting in unreasonable map pushing results, improves the accuracy of the map pushing result, and ensures the quality effect of the map pushing result. Therefore, it has good practical value.

[0093] In Figure 2 Based on the method shown, the embodiment of the present invention also provides another video processing method, which focuses on describing the process of selecting one or more target detection boxes corresponding to the target trajectory according to the congestion state corresponding to each segment of the target trajectory when the target object includes a local object of a specified target and an overall object of the specified target in the binding scenario; as Figure 4 shown, this method includes the following steps:

[0094] Step S402: Obtain the video stream for target tracking;

[0095] Step S404: Obtain the state parameters corresponding to at least one target trajectory in the video stream;

[0096] Among them, the state parameters are used to characterize the degree of proximity between the target trajectory and other trajectories in the video stream. The above target trajectory is the target trajectory corresponding to the local object of the specified target and the overall object of the specified target. In practical applications, the local object of the above specified target and the overall object of the specified target are in a bound relationship. Taking the specified target as a person as an example, the above local object can be a face or a head, and the overall object is a human body. If the vehicle is the specified target, the above local object can be a license plate or a logo, and the overall object is the vehicle body.

[0097] Step S406: Determine the segment to which the current video frame in the video stream belongs in the target trajectory according to the state parameters of the target trajectory;

[0098] Among them, the congestion states corresponding to adjacent segments of the target trajectory are different.

[0099] The above steps S402 to S406 can refer to the foregoing embodiments, and the embodiments of the present invention will not be elaborated herein.

[0100] Step S408: Determine the first target detection box corresponding to the local object and the second target detection box corresponding to the overall object according to the congestion states corresponding to each segment of the target trajectory; among them, the first target detection box and the second target detection box belong to the same video frame or different video frames;

[0101] Specifically, according to the congestion state, each segment can be divided into a congested segment and a non-congested segment. Therefore, the first target detection box corresponding to the local object and the second target detection box corresponding to the overall object can be determined from the congested segment and the non-congested segment respectively. One possible determination method includes: determining the non-congested segments according to the congestion states corresponding to each segment of the target trajectory; determining the first target detection box corresponding to the local object and the second target detection box corresponding to the overall object from the non-congested segments. In practical applications, for each non-congested segment, since the target object in the corresponding target detection box is in a non-congested scene with the remaining objects, the target detection box corresponding to the non-congested segment can ensure the image quality of the target object. Therefore, from the multiple candidate detection boxes corresponding to the non-congested segments, the first target detection box with better image quality of the local object and the second target detection box with better image quality of the overall object can be obtained, thereby ensuring the accuracy of the first target detection box corresponding to the local object and the second target detection box corresponding to the overall object, and further ensuring the accuracy of the map pushing result.

[0102] The above-mentioned crowded state corresponds to a crowding level, and the crowding level corresponds to the degree of crowding from crowded to sparse from high to low. Specifically, another possible determination method includes the following steps:

[0103] (1) For the first segment of the target trajectory with a crowding level greater than the level threshold, select the candidate detection frames corresponding to the local object and the global object respectively in each same video frame corresponding to the first segment;

[0104] For example, for the first segment that is relatively crowded, the candidate detection frames corresponding to the local object and the global object are determined through the same video frame to ensure that the candidate detection frames corresponding to the local object and the global object are the detection frames of the same specified target object, and avoid the binding error of the detection frames of the local object and the global object. For example, the local object is the face of target object A, while the global object is the body of target object B, etc.

[0105] (2) For the second segment of the target trajectory with a crowding level not greater than the level threshold, select the candidate detection frames corresponding to the local object and the global object respectively in different video frames corresponding to the second segment;

[0106] Specifically, the above-mentioned second segment is a not very crowded scene. The candidate detection frames corresponding to the local object and the global object can be determined by different video frames, that is, cross-frame. For example, the candidate detection frame corresponding to the global object such as the human body in the current video frame is relatively complete and clear, while the candidate detection frame corresponding to the local object such as the face in the next video frame of the current video frame is relatively complete and clear. At this time, the candidate detection frame corresponding to the global object is determined from the current video frame by cross-frame, and the candidate detection frame corresponding to the local object is determined from the next video frame respectively, thereby ensuring the accuracy of the target detection frame.

[0107] (3) Determine the first target detection frame corresponding to the local object and the second target detection frame corresponding to the global object according to the preset image quality weight and the crowding state weight corresponding to each segment.

[0108] Specifically, according to the preset image quality weight and the crowding state weight corresponding to each segment, the comprehensive score of the candidate detection frames corresponding to each crowded segment can be calculated, and the first target detection frame corresponding to the local object and the second target detection frame corresponding to the global object are determined from multiple candidate detection frames according to the comprehensive score, thereby ensuring the accuracy of the map pushing result. It should be noted that the first target detection frame and the second target detection frame determined by the comprehensive score of the candidate detection frames corresponding to each segment may belong to the same video frame or different video frames, and can be specifically set according to the actual situation. The embodiments of the present invention do not limit this.

[0109] Step S410: Use the first target detection box and the second target detection box as the target detection boxes of the specified target.

[0110] For the above video processing method, according to the congestion level, the same-frame image pushing strategy or the cross-frame image pushing strategy is flexibly applied to select the target detection boxes corresponding to the local object and the global object respectively in the video frames corresponding to each segment of the target trajectory, which can ensure that the target detection boxes of the global object and the local object are the detection boxes of the same specified object, thus alleviating the problem in the existing method that due to the ID Switch of the trajectory, the binding of the global object and the local object is incorrect, resulting in unreasonable image pushing results, improving the accuracy of the image pushing results, and ensuring the quality effect of the image pushing results. Therefore, it has good practical value.

[0111] Further, based on the method shown in Figure 2 the embodiments of the present invention also provide a method for obtaining state parameters, which focuses on describing the process of obtaining the state parameters corresponding to at least one target trajectory in the video stream. In practical applications, since the video stream includes multiple video frames, in the process of obtaining the state parameters corresponding to the target trajectory, the video frames in the video stream can be taken as the current video frame one by one according to the time sequence of the video frames in the video stream, and the method for obtaining the state parameters is executed to obtain the state parameters or the set of state parameters of the target trajectory in the video stream, which can be specifically set according to the actual situation.

[0112] As shown in Figure 5 the above method includes the following steps:

[0113] Step S502: Obtain the representative information of the target trajectory;

[0114] Among them, the above representative information is used to characterize the feature information and / or position information of the target object corresponding to the target trajectory; the position information includes the coordinate information of the prediction box of the target trajectory or the detection box corresponding to the target object in the previous frame of the current video frame; the above prediction box is the position box of the target object corresponding to the target trajectory predicted according to the position of the detection box in the existing video frames before the current video frame.

[0115] Specifically, for each target trajectory, the representative information of the target trajectory is calculated according to its detection box in the video frames of the video stream; among them, the representative information can be a representative box representing the features of the target object, or a list composed of multi-frame feature information of the target trajectory, or a fusion feature of multi-frame feature information. Therefore, the representative information is used to characterize the feature information and / or position information of the target object corresponding to the target trajectory.

[0116] In practical applications, since there is a detection box for the target object corresponding to each target trajectory in the video frame, in target tracking, the position box of the target object corresponding to the target trajectory in the current video frame can be predicted based on the positions of the detection boxes in the existing video frames before the current video frame. For example, the prediction boxes of the target objects corresponding to each target trajectory in the current video frame can be obtained through prediction methods such as linear prediction, quadratic prediction, and Kalman filter prediction. Optionally, the above representative information can be the coordinate information of the prediction box corresponding to the target trajectory, or the coordinate information of the detection box corresponding to the target object closest to the current video frame in the video stream, which can be specifically set according to the actual situation, and the embodiments of the present invention do not limit this.

[0117] Step S504: Determine the state parameters of the target trajectory corresponding to the current video frame according to the representative information and the detection box corresponding to the target object in the current video frame.

[0118] Specifically, the state parameters of the above target trajectory are not a single parameter value, but include multiple values corresponding one by one to the video frames; regarding the specific type of this value, the embodiments of the present invention do not limit this, as long as it can be used to characterize the proximity of the target trajectory to other trajectories in the current video frame. The above step S504 may include the following processes: (1) Obtain the target parameters corresponding to the representative information and the detection boxes corresponding to each target object in the current video frame; where the target parameters include: IoU and / or feature similarity; (2) Determine the state parameters of the target trajectory corresponding to the current video frame according to the representative box and the target parameters corresponding to each detection box.

[0119] The above IoU can be calculated based on the position information of the detection boxes corresponding to each target object in the current video frame and the representative information of the corresponding target trajectory. For the sake of understanding, here taking the representative information as the representative box of the target trajectory as an example, this representative box can be the prediction box of the target trajectory or the detection box corresponding to the target object in the previous frame of the current video frame. The representative box and the detection box A in the current video frame are as Figure 6 shown, where the shaded part M in 6-a represents the intersection of the representative box and the detection box A, and the dotted part N in 6-b represents the union of the representative box and the detection box A. Thus, according to the area sizes of M and N, the IoU of the representative box and the detection box A in the current video frame can be calculated, and the area sizes of M and N can be calculated according to the position information of the representative box and the detection box A, such as position coordinates.

[0120] The feature similarity can be calculated based on the feature vectors of the detection boxes in the current video frame and the feature information in the representative information of the corresponding target trajectory.

[0121] One possible way to determine the state parameter is to determine the state parameter of the target trajectory corresponding to the current video frame according to the representative box and the target parameters corresponding to each detection box, which can include one of the following four methods:

[0122] The first method: Count the number of target parameters corresponding to the representative information and each detection box that are greater than the first threshold to obtain a first value, and determine the state parameter of the target trajectory corresponding to the current video frame according to the first value. For example, use the first value as the state parameter of the target trajectory corresponding to the current video frame, etc.

[0123] The second method: Calculate the sum of the representative information and the target parameters corresponding to each detection box to obtain a second value, and determine the state parameter of the target trajectory corresponding to the current video frame according to the second value.

[0124] Specifically, taking the representative information as the representative box as an example, the IoU between the representative box of each target trajectory obtained by the above calculation and each detection box in the current video frame can be calculated, and the first threshold is set as the IoU threshold. Determine the size of each calculated IoU and the IoU threshold, and count the number of IoUs greater than the IoU threshold. Obtain the first value according to this number, and use the first value as the state parameter of the corresponding target trajectory in the current video frame; alternatively, the sum of all IoUs can also be calculated, and the second value is determined according to this sum, and the second value is used as the state parameter of the corresponding target trajectory in the current video frame.

[0125] Similarly, for the case where the target parameter is the feature similarity, or the target parameter is the IoU and the feature similarity, it is the same as the above IoU case, and the embodiments of the present invention will not be elaborated in detail here. It should be noted that for the sum of all the above IoUs, the larger the sum, the closer the target trajectory is to the remaining detection boxes in the current video frame except for the detection box of the corresponding target object, that is, at this time the target trajectory is prone to ID Switch. If the sum exceeds the preset sum threshold, the state corresponding to the state parameter of the target trajectory must be a crowded state.

[0126] The third method: (1) Determine the loss value between the representative information and each detection box according to the representative information and the target parameters corresponding to each detection box; wherein, the loss value is negatively correlated with the target parameter of the detection box; (2) Count the number of loss values between the representative information and each detection box that are less than the second threshold to obtain a third value, and determine the state parameter of the target trajectory corresponding to the current video frame according to the third value.

[0127] Specifically, taking the target parameter as IoU as an example, for the representative box of each target trajectory, by calculating the IoU between the representative box and each detection box in the current video frame, the loss value between the representative box and each detection box can be determined. For example, the loss value is 1 - IoU, etc. As long as the loss value is negatively correlated with the target parameter, and at this time, it is necessary to judge the number of loss values less than the second threshold, and use the determined third value as the state parameter of the target trajectory in the current video frame. It should be noted that for each existing loss value, the state parameter can be incremented by 1. Thus, according to the number of loss values less than the second threshold, the third value can be determined. The larger the third value, the more likely the target trajectory is to have an ID Switch. If the third value exceeds the preset third value threshold, the state corresponding to the state parameter of the target trajectory must be a crowded state.

[0128] In addition, in addition to counting the number of loss values less than the second threshold, the sum value of all loss values can also be calculated, and it is judged whether the sum value of the loss values is less than the loss value sum threshold. If it is less, the determined third value is used as the state parameter of the target trajectory in the current video frame. Similarly, for the case where the target parameter is feature similarity, or the target parameter is IoU and feature similarity, the process of determining the state parameter according to the loss value is the same as the above IoU case, and the embodiments of the present invention will not be elaborated in detail here.

[0129] The fourth method: (1) Determine the loss matrix according to the representative information of each trajectory and the target parameters corresponding to each detection box in the current video frame;

[0130] Among them, the elements in the loss matrix are the loss values corresponding to the trajectory and the detection box. Optionally, the row vector of the loss matrix is each trajectory, and the column vector is the loss value corresponding to each detection box. This loss value is negatively correlated with the target parameter corresponding to the detection box. Specifically, taking the target parameter as IoU as an example, the loss value is 1 - IoU. Then, for the 3 target trajectories Track1, Track2, and Track3 in the current video, the loss values of the corresponding representative boxes and the 4 detection boxes in the current video frame are calculated respectively, as shown in Table 2 below:

[0131] Table 2

[0132] Cost Mat det1 det2 det 3 det 4 Track1 0.77 0.97 0.45 1.00 Track2 0.13 0.84 0.69 1.00 Track3 0.82 0.25 0.93 1.00

[0133] Among them, det1 represents the loss value between detection box 1 and each target trajectory, det2 represents the loss value between detection box 2 and each target trajectory, det3 represents the loss value between detection box 3 and each target trajectory, and det4 represents the loss value between detection box 4 and each target trajectory.

[0134] (2) Assign trajectories to each detection box in the current video frame based on the loss matrix, and perform the following operations on the detection boxes with assigned trajectories: find the first loss value corresponding to the first detection box and the assigned first trajectory in the loss matrix, and the second loss value corresponding to the first detection box and other trajectories except the first trajectory; calculate the difference between the first loss value and the second loss value, if the absolute value of the difference is less than the third threshold, increment by 1 the statistical values corresponding to the first trajectory and other trajectories for the current video frame; determine the state parameter corresponding to the target trajectory for the current video frame according to the statistical value corresponding to the target trajectory at the end of the operation.

[0135] To ensure the minimum loss, the Hungarian algorithm can be used to assign each target trajectory. Among them, the larger the loss value, the smaller the IoU between the target trajectory and the detection box. A loss value of 1 means that the IoU between detection box 4 and the representative information of each target trajectory is 0, that is, detection box 4 has no relation with each target trajectory. Therefore, the assignment result is: detection box 3 is assigned to Track1, detection box  1 is assigned to Track2, and detection box 2 is assigned to Track3.

[0136] The detection box of the above-mentioned "assigned" trajectory means "based on the successfully matched trajectory". That is, for each successfully matched trajectory, consider the relationship between the loss value between this trajectory and the detection box it matches, and the loss value between this detection box and all other trajectories. Because when obtaining the matching result for the loss matrix based on the Hungarian algorithm, the matching is performed according to the principle of minimum loss. Even if there is no intersection between each trajectory and each detection box, or the intersection is very small, the Hungarian algorithm will give a matching result, and then it can be filtered according to the specified threshold. For example, assume that there is only one detection box and only one trajectory in the current video frame, but their IoU is 0. At this time, the Hungarian algorithm will match this detection box and this trajectory together. However, through the filtering of the specified threshold of IoU, for example, the specified threshold corresponding to IoU is 0.05, then it is required that the IoU between the representative box of this trajectory and this detection box is greater than 0.05 to be considered a successful match. Since the above-mentioned IoU is 0, it can be filtered out by setting a threshold (such as 0.05).

[0137] In addition, for the detection box 3, the first loss value corresponding to the matched Track1 is 0.45, and the second loss value corresponding to Track2 is 0.69. Since the first loss value and the second loss value are relatively close, the difference between the first loss value and the second loss value can be calculated, and it is determined that the absolute value of the difference is less than the third threshold. If it is less, it means that the first loss value and the second loss value are very close, and the possibility of ID Switch occurring between Track1 and Track2 is increased by 1. Therefore, at this time, the statistical values of Track1 and Track2 corresponding to the current video frame are incremented by 1; the statistical values of Track1 and Track2 corresponding to the current video frame at the end of the operation are used as the state parameters of Track1 and Track2 corresponding to the current video frame.

[0138] In addition, for the above loss matrix, the state parameters can also be directly calculated; for example, for the loss matrix in Table 2 above, the rows of the loss matrix can also be summed, such as calculating the sum of the loss values of Track1 and the 4 detection boxes, to characterize the overlapping degree of the target trajectory Track1 and all detection boxes in the current video frame. If the calculated row sum value is less than the preset row sum value threshold, it is considered that Track1 is in a crowded state, and the current video frame is the segment corresponding to the crowded state; or the statistical value of Track1 corresponding to the current video frame can also be incremented by 1 according to the row sum value being less than the preset row sum value threshold, and the statistical value of Track1 corresponding to the current video frame at the end is used as the state parameter of Track1 corresponding to the current video frame.

[0139] Since the state parameters are the states of each target trajectory corresponding to different video frames, such as crowded state and non-crowded state, etc., the target trajectories can be divided temporally through the state variables. Since ID Switch is likely to occur in the crowded state, when a certain target trajectory moves from a sparse area to a dense area and then to a sparse area again, the target trajectory can be divided into the first segment corresponding to the sparse state, the second segment corresponding to the crowded state, and the third segment corresponding to the sparse state. At the same time, it can also be considered that the target trajectory is divided into a segment with a low probability of ID Switch, a segment with a high probability of ID Switch, and a segment with a low probability of ID Switch. By calculating the state parameters for the segment with a high probability of ID Switch, this segment can be further refined into multiple segments temporally according to the degree of crowding or density. Therefore, compared with the existing methods, for dense or crowded scenarios, the accuracy of the map inference result is ensured by determining the state parameters of the target trajectory corresponding to the current video frame.

[0140] Therefore, the method for obtaining the above state parameters calculates the representative information of each target trajectory and the target parameters of each detection box in the current video frame, and determines the state parameters of the target trajectory corresponding to the current video frame based on the target parameters, which can make the determined state parameters more reasonable. Based on this method for obtaining state parameters, it can be ensured that the selected target detection boxes are more reasonable.

[0141] Next, taking the application scenario of human and face binding as an example, refer to Figure 7 the video processing method shown. This method focuses on describing the process of pushing images in the binding scenario. Here, pushing images has the same meaning as selecting one or more target detection boxes. Specifically, this method includes the following steps:

[0142] Step S702, use a real-time tracking algorithm (such as IoU Tracker) to assign each detection box in the current frame to the corresponding tracking trajectory.

[0143] The detection boxes in this embodiment include the detection boxes corresponding to the human face and the human body respectively.

[0144] Step S704, calculate the state parameters of each tracking trajectory corresponding to this current frame.

[0145] The above state parameters indicate whether the state of the tracking trajectory at different times is prone to ID Switch. Therefore, the scene where the target object is located can be estimated based on the state reference. The method for calculating the state parameters can refer to the above embodiment. For example, for IoU Tracker, a simple instance state calculation method is to count the IoU situation between the detection boxes of the target trajectory and the prediction boxes of other trajectories in the loss matrix (directly sum the IoUs or record as 1 if below a specified threshold and then accumulate). The larger the state parameter, the more likely the target object is in a crowded scene, and thus this trajectory is prone to ID Switch.

[0146] Through the above calculation, the state parameters of each trajectory corresponding to different frames, that is, different times, are obtained. Taking a certain trajectory as an example, the state parameters of the trajectory corresponding to different times can be recorded in the following way: Track(0) = [box(0), state(0)], Track(1) = [box(1), state(1)], Track(2) = [box(2), state(2)], ……, Track(t) = [box(t), state(t)]. Where t is the time identifier, box is the detection box identifier, and state is the state parameter.

[0147] Step S706, divide the tracking trajectories in chronological order according to their state parameters to obtain each segment of the tracking trajectories.

[0148] For example, when a certain trajectory enters a dense area from a sparse area and then enters a sparse area again, it is considered that the trajectory should be divided into three segments, corresponding to the low-occurrence segment of ID Switch, the high-occurrence segment of ID Switch, and the low-occurrence segment of ID Switch. Note that the trajectory segmentation is calculated based on the distribution of the state parameters of the tracked trajectory in this segment. For example, if the state reference of the trajectory in the current trajectory segment is always greater than the specified threshold, it is considered that this trajectory segment is in the high-occurrence state segment of ID Switch.

[0149] Taking Track(0) = [box(0), state(0)], Track(1) = [box(1), state(1)], Track(2) = [box(2), state(2)], ……, Track(t) = [box(t), state(t)] as an example, assuming that the segmentation method is to compare the state at the current moment with the state at the previous moment, and start segmenting this trajectory from t = 0. If the current state is equal to the state at the previous moment or the difference is within the set threshold, the current moment and the previous moment can be divided into the same segment. If the current state is not equal to the state at the previous moment or the difference exceeds the set threshold, the current moment is a new segment.

[0150] Step S708, perform alternative map inference in each segment of the tracked trajectory.

[0151] This segmented method for face and body map inference can avoid the phenomenon of incorrect binding of faces and bodies. For example, when a certain trajectory passes through the high-occurrence state of ID Switch, there is a high probability that the segments before and after this moment belong to two different trajectories. If map inference is directly performed, it is easy to have the situation where the face and body map inferences belong to different instances. However, performing face and body map inference separately for each segment can avoid this problem.

[0152] Step S710, based on the alternative map inferences of each segment of the tracked trajectory, select the best one to determine the final map inference.

[0153] For example, it can be considered to select the best face among all the alternative map inferences, and the alternative body in the segment where this face is located, and combine them into (face map inference, body map inference) as the final result. Or it can be considered to select the best face among all the alternative map inferences, and the best body among all the alternative map inferences, and combine them into (face map inference, body map inference) as the final result.

[0154] For the binding scenario, the above-mentioned map inference method based on state parameters can not only improve the binding accuracy, but also improve the map inference quality. For example, an intuitive method to improve the binding accuracy is to directly check which frame contains the optimal face, and then infer the face and the human body in the same frame. However, in this case, the quality of human body map inference will be significantly reduced. After introducing the map inference strategy based on state parameters, for some trajectories that have always been in a "sparse" state and only contain one target (such as a person), there is only one segment in the whole trajectory. Then, if the optimal face and the optimal human body are selected within this segment as the final map inference (the optimal face and human body map inferences do not necessarily appear in the same frame), the map inference quality can be guaranteed. In the prior art, the same-frame map Figure 1 is generally to calculate the quality by weighting the face and the human body, and then determine the map inference frame, so there may be problems such as a decrease in the quality of the human body or the face.

[0155] Corresponding to the above video processing method, an embodiment of the present invention also provides a video processing device. As Figure 8 shown, the device includes a video stream acquisition module 81, a state parameter acquisition module 82, and a detection box selection module 83 that are connected in sequence; among them, the functions of each module are as follows:

[0156] The video stream acquisition module 81 is used to acquire a video stream for target tracking;

[0157] The state parameter acquisition module 82 is used to acquire state parameters corresponding to at least one target trajectory in the video stream; wherein, the state parameters are used to characterize the proximity of the target trajectory to other trajectories in the video stream;

[0158] The detection box selection module 83 is used to select one or more target detection boxes according to the state parameters of the target trajectory.

[0159] The video processing device provided by the embodiment of the present invention can select one or more target detection boxes, that is, the map inference result, by acquiring the state parameters corresponding to the target trajectory in the video stream. This video processing method that considers the state variables of the proximity of the target trajectory to other trajectories in the video stream during the map inference process can make the factors considered in the map inference more comprehensive, thereby improving the rationality of the map inference result, effectively ensuring the quality of trajectory map inference, and having good practical value.

[0160] In one possible embodiment, the above state parameter acquisition module 82 is further used to: determine the segment to which the current video frame of the video stream belongs in the target trajectory according to the state parameters of the target trajectory; wherein, the congestion states corresponding to adjacent segments of the target trajectory are different; select one or more target detection boxes corresponding to the target trajectory according to the congestion states corresponding to each segment of the target trajectory.

[0161] In another possible embodiment, selecting one or more target detection boxes corresponding to the target trajectory according to the congestion states corresponding to each segment of the target trajectory includes: smoothing the congestion states corresponding to each segment of the target trajectory; and selecting one or more target detection boxes corresponding to the target trajectory according to the smoothed congestion states corresponding to each segment of the target trajectory.

[0162] In another possible embodiment, selecting one or more target detection boxes corresponding to the target trajectory according to the congestion states corresponding to each segment of the target trajectory includes: obtaining candidate detection boxes corresponding to each segment of the target trajectory for the target object; and determining one or more target detection boxes of the target trajectory according to a preset image quality weight and / or a congestion state weight corresponding to each segment.

[0163] In another possible embodiment, the target object includes a local object of a specified target and an overall object of the specified target; selecting one or more target detection boxes corresponding to the target trajectory according to the congestion states corresponding to each segment of the target trajectory includes: determining a first target detection box corresponding to the local object and a second target detection box corresponding to the overall object according to the congestion states corresponding to each segment of the target trajectory; wherein the first target detection box and the second target detection box belong to the same video frame or different video frames; and using the first target detection box and the second target detection box as the target detection boxes of the specified target.

[0164] In another possible embodiment, determining a first target detection box corresponding to the local object and a second target detection box corresponding to the overall object according to the congestion states corresponding to each segment of the target trajectory includes: determining non-congested segments according to the congestion states corresponding to each segment of the target trajectory; and determining the first target detection box corresponding to the local object and the second target detection box corresponding to the overall object from the non-congested segments.

[0165] In another possible embodiment, the congestion state corresponds to a congestion level, and the congestion level corresponds to the congestion degree from congestion to sparsity from high to low; determining a first target detection box corresponding to the local object and a second target detection box corresponding to the overall object according to the congestion states corresponding to each segment of the target trajectory includes: for a first segment of the target trajectory with a congestion level greater than a level threshold, selecting candidate detection boxes corresponding to the local object and the overall object respectively in each of the same video frames corresponding to the first segment; for a second segment of the target trajectory with a congestion level not greater than the level threshold, selecting candidate detection boxes corresponding to the local object and the overall object respectively in different video frames corresponding to the second segment; and determining the first target detection box corresponding to the local object and the second target detection box corresponding to the overall object according to a preset image quality weight and the congestion state weights corresponding to each segment.

[0166] In another possible embodiment, the specified target is a person, the local object is a face, and the overall object is a human body; or, the specified target is a vehicle, the local object is a license plate, and the overall object is a vehicle body.

[0167] In another possible embodiment, the above-mentioned state parameter acquisition module 82 is further configured to: sequentially take each video frame in the video stream as the current video frame according to the time sequence of the video frames in the video stream, and perform the following operations: obtain the representative information of the target trajectory; wherein, the representative information is used to characterize the feature information and / or position information of the target object corresponding to the target trajectory; the position information includes the prediction box of the target trajectory or the coordinate information of the detection box corresponding to the target object in the previous frame of the current video frame; wherein, the prediction box is the position box of the target object corresponding to the target trajectory predicted in the current video frame according to the positions of the detection boxes in the video frames existing before the current video frame; determine the state parameter of the target trajectory corresponding to the current video frame according to the representative information and the detection box corresponding to the target object in the current video frame.

[0168] In another possible embodiment, the state parameters of the above-mentioned target trajectory include a plurality of values corresponding one by one to the video frames; the determining the state parameter of the target trajectory corresponding to the current video frame according to the representative information and the detection box corresponding to the target object in the current video frame includes: obtaining the target parameters corresponding to the representative information and the detection boxes corresponding to each target object in the current video frame; wherein, the target parameters include: IoU and / or feature similarity; determine the state parameter of the target trajectory corresponding to the current video frame according to the representative information and the target parameters corresponding to each detection box.

[0169] In another possible embodiment, the determining the state parameter of the target trajectory corresponding to the current video frame according to the representative information and the target parameters corresponding to each detection box includes: counting the number of target parameters corresponding to the representative information and each detection box that are greater than the first threshold to obtain a first value, and determining the state parameter of the target trajectory corresponding to the current video frame according to the first value; or, calculating the sum of the representative information and the target parameters corresponding to each detection box to obtain a second value, and determining the state parameter of the target trajectory corresponding to the current video frame according to the second value.

[0170] In another possible embodiment, the determining the state parameter of the target trajectory corresponding to the current video frame according to the representative information and the target parameters corresponding to each detection box includes: determining the loss values of the representative information and each detection box according to the representative information and the target parameters corresponding to each detection box; wherein, the loss value is negatively correlated with the target parameter of the detection box; counting the number of loss values of the representative information and each detection box that are less than the second threshold to obtain a third value, and determining the state parameter of the target trajectory corresponding to the current video frame according to the third value.

[0171] In another possible embodiment, determining the state parameter of the target trajectory corresponding to the current video frame according to the representative information and the target parameters corresponding to each detection box includes: determining a loss matrix according to the representative information of each trajectory and the target parameters corresponding to each detection box in the current video frame; wherein, the elements in the loss matrix are the loss values corresponding to the trajectory and the detection box, and the loss value is negatively correlated with the target parameter corresponding to the detection box; based on the loss matrix, allocating trajectories to each detection box in the current video frame, and performing the following operations on the detection boxes with allocated trajectories: finding the first loss value corresponding to the first detection box and the first trajectory allocated from the loss matrix, and the second loss value corresponding to the first detection box and other trajectories except the first trajectory; calculating the difference between the first loss value and the second loss value, if the absolute value of the difference is less than the third threshold, adding 1 to the statistical values corresponding to the first trajectory and other trajectories in the current video frame; determining the state parameter of the target trajectory corresponding to the current video frame according to the statistical value of the target trajectory corresponding to the current video frame at the end of the operation.

[0172] In another possible embodiment, the step of determining the segment to which the current video frame of the video stream belongs in the target trajectory according to the state parameter of the target trajectory includes: segmenting the current video frame corresponding to the target trajectory according to the state parameter of the target trajectory; wherein, the segmentation includes at least one crowded segment and / or at least one non-crowded segment.

[0173] In another possible embodiment, segmenting the current video frame corresponding to the target trajectory according to the state parameter of the target trajectory includes one of the following: determining the segment to which the current video frame corresponding to the target trajectory belongs according to the current state parameter of the target trajectory; in the existing video frames corresponding to the target trajectory, calculating the reference value of the state parameters of a specified number of consecutive video frames starting from the current video frame, and determining the segment to which the current video frame belongs according to the reference value; wherein, the reference value includes the sum value or the average value; updating the cumulative statistic according to the current state parameter of the target trajectory, and determining the segment to which the current video frame corresponding to the target trajectory belongs according to the updated cumulative statistic; wherein, the cumulative statistic is used to characterize the crowded state of the most recent segment of the target trajectory.

[0174] The video processing device provided by the embodiments of the present invention has the same implementation principle and the same technical effects as those of the foregoing video processing method embodiments. For the sake of brief description, for the parts not mentioned in the embodiments of the video processing device, reference may be made to the corresponding content in the foregoing video processing method embodiments.

[0175] Embodiments of the present invention also provide an electronic device, such as Figure 9As shown, it is a schematic structural diagram of the electronic device. Among them, the electronic device includes a processor 91 and a memory 92. The memory 92 stores machine-executable instructions that can be executed by the processor 91. The processor 91 executes the machine-executable instructions to implement the above video processing method.

[0176] In Figure 9 the illustrated embodiment, the electronic device further includes a bus 93 and a communication interface 94. Among them, the processor 91, the communication interface 94, and the memory 92 are connected through the bus.

[0177] Among them, the memory 92 may include a high-speed random access memory (RAM, Random Access Memory), and may also include a non-volatile memory, such as at least one disk memory. Through at least one communication interface 94 (which can be wired or wireless), a communication connection is realized between the system network element and at least one other network element. The Internet, wide area network, local area network, metropolitan area network, etc. can be used. The bus can be an ISA (Industry Standard Architecture, industrial standard architecture) bus, a PCI (Peripheral Component Interconnect, peripheral component interconnect standard) bus, or an EISA (Extended Industry Standard Architecture, extended industrial standard structure) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of representation, Figure 9 only a bidirectional arrow is used in

[0178] The processor 91 may be an integrated circuit chip with the ability to process signals. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in the processor 91 or the instructions in the form of software. The above-mentioned processor 91 may be a general-purpose processor, including a central processing unit (CPU for short), a network processor (NP for short), etc.; it may also be a digital signal processor (DSP for short), an application specific integrated circuit (ASIC for short), a field-programmable gate array (FPGA for short), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. It can implement or execute each method, step and logic block diagram disclosed in the embodiments of the present invention. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in combination with the embodiments of the present invention can be directly embodied as being executed and completed by the hardware decoding processor, or executed and completed by a combination of the hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. This storage medium is located in the memory, and the processor 91 reads the information in the memory 92 and combines its hardware to complete the steps of the video processing method in the foregoing embodiments.

[0179] The embodiment of the present invention also provides a machine-readable storage medium, which stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions cause the processor to implement the above video processing method. For the specific implementation, reference can be made to the foregoing method embodiments, and details are not described herein again.

[0180] The computer program product of the video processing method, device and electronic system provided by the embodiment of the present invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the video processing method described in the foregoing method embodiments. For the specific implementation, reference can be made to the method embodiments, and details are not described herein again.

[0181] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the systems and devices described above can refer to the corresponding processes in the foregoing method embodiments, and details are not described herein again.

[0182] In addition, in the description of the embodiments of the present invention, unless otherwise clearly defined and limited, the terms "installation", "connection", and "coupling" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium, and it can be the communication inside two components. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0183] If the above-mentioned function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a non-volatile computer-readable storage medium executable by a processor. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories, random access memories, magnetic disks, or optical discs that can store program codes.

[0184] In the description of the present invention, it should be noted that the orientation or positional relationship indicated by the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc. is based on the orientation or positional relationship shown in the drawings. It is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it should not be construed as a limitation to the present invention. In addition, the terms "first", "second", and "third" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0185] Finally, it should be noted that the above-mentioned embodiments are only specific implementation manners of the present invention, used to illustrate the technical solutions of the present invention, rather than limiting it. The protection scope of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: any person skilled in the art within the technical scope disclosed by the present invention can still modify the technical solutions described in the foregoing embodiments, or can easily think of changes, or perform equivalent replacements on some of the technical features; and these modifications, changes, or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered by the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims

1. A video processing method, characterized in that, The method includes: Obtaining a video stream for target tracking; Obtaining state parameters corresponding to at least one target trajectory in the video stream; wherein, the state parameters are used to characterize the proximity of the target trajectory to other trajectories in the video stream; Determining, according to the state parameters of the target trajectory, the segment to which the current video frame of the video stream belongs in the target trajectory; wherein, the congestion states corresponding to adjacent segments of the target trajectory are different; Selecting one or more target detection frames corresponding to the target trajectory according to the congestion states corresponding to each segment of the target trajectory; The target object includes a local object of a specified target and an overall object of the specified target; the step of selecting one or more target detection frames corresponding to the target trajectory according to the congestion states corresponding to each segment of the target trajectory includes: Determining a first target detection frame corresponding to the local object and a second target detection frame corresponding to the overall object according to the congestion states corresponding to each segment of the target trajectory; wherein, the first target detection frame and the second target detection frame belong to the same video frame or different video frames; Taking the first target detection frame and the second target detection frame as the target detection frames of the specified target; The congestion state corresponds to a congestion level, and the congestion level corresponds to the congestion degree from high to low in order of decreasing congestion; the step of determining a first target detection frame corresponding to the local object and a second target detection frame corresponding to the overall object according to the congestion states corresponding to each segment of the target trajectory includes: for a first segment of the target trajectory with a congestion level greater than a level threshold, selecting candidate detection frames corresponding to the local object and the overall object respectively in each same video frame corresponding to the first segment; for a second segment of the target trajectory with a congestion level not greater than the level threshold, selecting candidate detection frames corresponding to the local object and the overall object respectively in different video frames corresponding to the second segment; determining the first target detection frame corresponding to the local object and the second target detection frame corresponding to the overall object according to a preset image quality weight and a congestion state weight corresponding to each segment.

2. The method according to claim 1, wherein The step of selecting one or more target detection frames corresponding to the target trajectory according to the congestion states corresponding to each segment of the target trajectory includes: Performing smoothing processing on the congestion states corresponding to each segment of the target trajectory; Selecting one or more target detection frames corresponding to the target trajectory according to the smoothed congestion states corresponding to each segment of the target trajectory.

3. The method according to claim 2, wherein The step of selecting one or more target detection frames corresponding to the target trajectory according to the congestion states corresponding to each segment of the target trajectory includes: Obtaining candidate detection frames corresponding to each segment of the target trajectory for the target object; Determining one or more target detection frames of the target trajectory according to a preset image quality weight and / or a congestion state weight corresponding to each segment.

4. The method according to claim 3, characterized in that The step of determining a first target detection box corresponding to the local object and a second target detection box corresponding to the overall object according to the congestion states corresponding to the respective segments of the target trajectory includes: Determine non-congested segments according to the congestion states corresponding to the respective segments of the target trajectory; Determine a first target detection box corresponding to the local object and a second target detection box corresponding to the overall object from the non-congested segments.

5. The method according to claim 3, characterized in that, The specified target is a person, the local object is a face, and the overall object is a human body; Alternatively, the specified target is a vehicle, the local object is a license plate, and the overall object is a vehicle body.

6. The method according to any one of claims 1-5, characterized in that The step of obtaining state parameters corresponding to at least one target trajectory in the video stream includes: Sequentially take each video frame in the video stream as the current video frame according to the time sequence of the video frames in the video stream, and perform the following operations: Obtain representative information of the target trajectory; wherein, the representative information is used to characterize the feature information and / or position information of the target object corresponding to the target trajectory; the position information includes the predicted box of the target trajectory or the coordinate information of the detection box corresponding to the target object in the previous frame of the current video frame; wherein, the predicted box is the position box of the target object corresponding to the target trajectory predicted according to the position of the detection box of the target trajectory in the video frames existing before the current video frame in the current video frame; Determine the state parameter corresponding to the current video frame of the target trajectory according to the representative information and the detection box corresponding to the target object in the current video frame.

7. The method according to claim 6, wherein The state parameters of the target trajectory include a plurality of values corresponding one by one to the video frames; The step of determining the state parameter corresponding to the current video frame of the target trajectory according to the representative information and the detection box corresponding to the target object in the current video frame includes: Obtain target parameters corresponding to the representative information and the detection boxes corresponding to the respective target objects in the current video frame; wherein, the target parameters include: IoU and / or feature similarity; Determine the state parameter corresponding to the current video frame of the target trajectory according to the representative information and the target parameters corresponding to the respective detection boxes.

8. The method according to claim 7, wherein The step of determining the state parameter corresponding to the current video frame of the target trajectory according to the representative information and the target parameters corresponding to the respective detection boxes includes: Count the number of target parameters corresponding to the representative information and the respective detection boxes that are greater than a first threshold to obtain a first value, and determine the state parameter corresponding to the current video frame of the target trajectory according to the first value; or, Calculate the sum of the representative information and the target parameters corresponding to the respective detection boxes to obtain a second value, and determine the state parameter corresponding to the current video frame of the target trajectory according to the second value.

9. The method according to claim 7, wherein The step of determining the state parameter corresponding to the current video frame of the target trajectory according to the representative information and the target parameters corresponding to the respective detection boxes includes: Determine the loss values of the representative information and each of the detection boxes based on the representative information and the target parameters corresponding to each of the detection boxes; wherein, the loss value is negatively correlated with the target parameters of the detection box. Count the number of loss values of the representative information and each of the detection boxes that are less than a second threshold to obtain a third value, and determine the state parameter of the target trajectory corresponding to the current video frame according to the third value.

10. The method according to claim 7, characterized in that, The step of determining the state parameter of the target trajectory corresponding to the current video frame according to the representative information and the target parameters corresponding to each of the detection boxes includes: Determine a loss matrix according to the representative information of each trajectory and the target parameters corresponding to each detection box in the current video frame; wherein, the elements in the loss matrix are the loss values corresponding to the trajectory and the detection box, and the loss value is negatively correlated with the target parameters of the detection box. Based on the loss matrix, assign trajectories to each detection box in the current video frame, and perform the following operations on the detection boxes with assigned trajectories: Search in the loss matrix for the first loss value of the first detection box corresponding to the assigned first trajectory and the second loss value of the first detection box corresponding to other trajectories except the first trajectory. Calculate the difference between the first loss value and the second loss value. If the absolute value of the difference is less than a third threshold, increment by 1 the statistical values of the first trajectory and the other trajectories corresponding to the current video frame. Determine the state parameter of the target trajectory corresponding to the current video frame according to the statistical value of the target trajectory corresponding to the current video frame when the operation ends.

11. The method according to claim 1, characterized in that, The step of determining the segment to which the current video frame of the video stream belongs in the target trajectory according to the state parameter of the target trajectory includes: Segment the current video frame corresponding to the target trajectory according to the state parameter of the target trajectory; wherein, the segment includes at least one crowded segment and / or at least one non-crowded segment.

12. The method according to claim 11, wherein The step of segmenting the current video frame corresponding to the target trajectory according to the state parameter of the target trajectory includes one of the following: Determine the segment to which the current video frame corresponding to the target trajectory belongs according to the current state parameter of the target trajectory. In the existing video frames corresponding to the target trajectory, calculate the reference value of the state parameters of a specified number of consecutive video frames starting from the current video frame. Determine the segment to which the current video frame belongs according to the reference value; wherein, the reference value includes a sum value or an average value. Update the cumulative statistic according to the current state parameter of the target trajectory, and determine the segment to which the current video frame corresponding to the target trajectory belongs according to the updated cumulative statistic; wherein, the cumulative statistic is used to characterize the crowded state of the most recent segment of the target trajectory.

13. A video processing device, characterized in that, The device includes: A video stream acquisition module, configured to acquire a video stream for target tracking. A state parameter acquisition module, configured to acquire the state parameters corresponding to at least one target trajectory in the video stream; wherein, the state parameter is used to characterize the proximity of the target trajectory to other trajectories in the video stream. The detection box selection module is configured to: determine the segment to which the current video frame of the video stream belongs in the target trajectory according to the state parameters of the target trajectory; wherein, the congestion states corresponding to adjacent segments of the target trajectory are different; select one or more target detection boxes corresponding to the target trajectory according to the congestion states corresponding to each segment of the target trajectory; The target object includes a local object of a specified target and an overall object of the specified target; the selecting one or more target detection boxes corresponding to the target trajectory according to the congestion states corresponding to each segment of the target trajectory includes: determining a first target detection box corresponding to the local object and a second target detection box corresponding to the overall object according to the congestion states corresponding to each segment of the target trajectory; wherein, the first target detection box and the second target detection box belong to the same video frame or different video frames; using the first target detection box and the second target detection box as the target detection boxes of the specified target; The congestion state corresponds to a congestion level, and the congestion level corresponds to the congestion degree from congestion to sparsity from high to low; the determining a first target detection box corresponding to the local object and a second target detection box corresponding to the overall object according to the congestion states corresponding to each segment of the target trajectory includes: for a first segment of the target trajectory with a congestion level greater than a level threshold, selecting candidate detection boxes corresponding to the local object and the overall object respectively in each same video frame corresponding to the first segment; for a second segment of the target trajectory with a congestion level not greater than the level threshold, selecting candidate detection boxes corresponding to the local object and the overall object respectively in different video frames corresponding to the second segment; determining the first target detection box corresponding to the local object and the second target detection box corresponding to the overall object according to a preset image quality weight and a congestion state weight corresponding to each segment.

14. An electronic system, characterized in that, The electronic system includes: an image acquisition device, a processing device, and a storage device; The image acquisition device is configured to acquire a video stream; A computer program is stored on the storage device, and the computer program, when run by the processing device, executes the video processing method according to any one of claims 1 to 12.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is run by the processing device, it executes the steps of the video processing method according to any one of claims 1 to 12 above.

Images