Data processing method and device, image processing method and device
By obtaining and segmenting video frames in video processing, identifying the target external frame and action trajectory of the object, the problem of difficulty in identifying target objects in the video is solved, and the accurate tracking and recognition of the protagonists in the video is achieved, and the efficiency of video processing is improved. and accuracy.
Patent Information
- Application Number
- CN201911395402.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2019-12-30
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2039-12-30
AI Technical Summary
The prior art is difficult to effectively determine the target object in a video, especially when multiple objects exist in a video frame, and it is difficult to accurately track and identify the action trajectory of the protagonist, affecting the accuracy and efficiency of video processing.
By obtaining the pending video, segmenting the video frames, obtaining the object's target external boxes, and determining the object's action trajectory and action trajectory sequence based on these external boxes, we can then identify the target object in the video, improving the robustness and target of the action trajectory tracking and improving the The accuracy of the object.
It realizes accurate identification and tracking of target objects in the video, improves the accuracy and efficiency of video processing, can better meet users' attention needs for the protagonist, and improves the ratings.
Smart Images

Figure CN113066101B_ABST
Abstract
Description
Technical Field
[0001] Embodiments of this specification relate to the field of video processing technology, and more particularly to a data processing method and an image processing method. One or more embodiments of this specification also relate to a data processing device, an image processing device, an asset valuation method, an asset valuation device, three computing devices, and two computer-readable storage media. Background Art
[0002] Generally speaking, when watching a video, users tend to pay more attention to the protagonist in the video, and users will pay more attention to videos containing the protagonist. Therefore, extracting the video frames containing the protagonist and then pushing them to users for viewing can achieve better viewing effects.
[0003] Before extracting the video frames containing the protagonist in a video, the protagonist in the video must first be determined. Therefore, there is an urgent need to provide a method for obtaining the target object in the video. Summary of the Invention
[0004] In view of this, the present invention provides a data processing method and an image processing method. One or more embodiments of the present invention also provide a data processing device, an image processing device, an asset valuation method, an asset valuation device, three computing devices, and two computer-readable storage media to address technical deficiencies in the prior art.
[0005] According to a first aspect of an embodiment of this specification, there is provided a data processing method, including:
[0006] Get the video to be processed;
[0007] Segmenting the video to be processed to obtain one or more original video frames;
[0008] Obtaining target bounding boxes of all objects in the one or more original video frames, and determining motion trajectories of all objects based on the target bounding boxes;
[0009] A target bounding box sequence corresponding to the motion trajectories of all objects is obtained, and a target object in the video is determined based on the target bounding box sequence.
[0010] According to a second aspect of the embodiments of this specification, there is provided an image processing method, including:
[0011] Acquire a video to be processed, wherein the video includes a video frame sequence;
[0012] Obtaining all video frames of the video frame sequence, wherein the video frames include one or more objects;
[0013] Obtaining a target bounding box of the one or more objects;
[0014] determining a motion trajectory of the one or more objects based on the target bounding box;
[0015] Obtaining a target bounding box sequence corresponding to the motion trajectory of the one or more objects;
[0016] A target object in the video is determined based on the target bounding box sequence.
[0017] According to a third aspect of the embodiments of this specification, there is provided a data processing device, including:
[0018] A demand acquisition module is configured to acquire the video to be processed;
[0019] A video segmentation module is configured to segment the video to be processed to obtain one or more original video frames;
[0020] a motion trajectory acquisition module configured to acquire target bounding boxes of all objects in the one or more original video frames, and determine motion trajectories of all objects based on the target bounding boxes;
[0021] The object acquisition module is configured to acquire a target circumscribed frame sequence corresponding to the motion trajectories of all objects, and determine a target object in the video based on the target circumscribed frame sequence.
[0022] According to a fourth aspect of the embodiments of this specification, there is provided an image processing apparatus, including:
[0023] A video acquisition module is configured to acquire a video to be processed, wherein the video includes a video frame sequence;
[0024] a video frame obtaining module, configured to obtain all video frames of the video frame sequence, wherein the video frames include one or more objects;
[0025] An external frame acquisition module is configured to acquire a target external frame of the one or more objects;
[0026] a trajectory determination module, configured to determine the motion trajectory of the one or more objects based on the target bounding box;
[0027] an external frame sequence acquisition module, configured to acquire a target external frame sequence corresponding to the motion trajectory of the one or more objects;
[0028] The target object determination module is configured to determine the target object in the video based on the target circumscribed frame sequence.
[0029] According to a fifth aspect of the embodiments of this specification, there is provided a computing device, including:
[0030] memory and processor;
[0031] The memory is configured to store computer-executable instructions, and the processor is configured to execute the computer-executable instructions:
[0032] Get the video to be processed;
[0033] Segmenting the video to be processed to obtain one or more original video frames;
[0034] Obtaining target bounding boxes of all objects in the one or more original video frames, and determining motion trajectories of all objects based on the target bounding boxes;
[0035] A target bounding box sequence corresponding to the motion trajectories of all objects is obtained, and a target object in the video is determined based on the target bounding box sequence.
[0036] According to a sixth aspect of the embodiments of this specification, there is provided a computing device, including:
[0037] memory and processor;
[0038] The memory is configured to store computer-executable instructions, and the processor is configured to execute the computer-executable instructions:
[0039] Acquire a video to be processed, wherein the video includes a video frame sequence;
[0040] Obtaining all video frames of the video frame sequence, wherein the video frames include one or more objects;
[0041] Obtaining a target bounding box of the one or more objects;
[0042] determining a motion trajectory of the one or more objects based on the target bounding box;
[0043] Obtaining a target bounding box sequence corresponding to the motion trajectory of the one or more objects;
[0044] A target object in the video is determined based on the target bounding box sequence.
[0045] According to a seventh aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions, which implement the steps of the data processing method or the image processing method when executed by a processor.
[0046] According to an eighth aspect of the embodiments of this specification, there is provided an asset evaluation method, comprising:
[0047] Obtaining a video to be processed, and segmenting the video to be processed to obtain one or more original video frames;
[0048] Obtaining target bounding boxes of all objects in the one or more original video frames, and determining motion trajectories of all objects based on the target bounding boxes;
[0049] Obtaining a target bounding box sequence corresponding to the motion trajectories of all objects, and determining an area value of each object in the video to be processed based on the target bounding box sequence;
[0050] The asset value of all objects is evaluated based on the area value of each object and the preset asset value.
[0051] According to a ninth aspect of the embodiments of this specification, there is provided an asset evaluation device, comprising:
[0052] The video frame obtaining device is configured to obtain a video to be processed and segment the video to be processed to obtain one or more original video frames;
[0053] a trajectory determination device configured to obtain target bounding boxes of all objects in the one or more original video frames, and determine the motion trajectories of all objects based on the target bounding boxes;
[0054] an area value determining device configured to obtain a target circumscribed frame sequence corresponding to the motion trajectories of all objects, and determine an area value of each object in the video to be processed based on the target circumscribed frame sequence;
[0055] The evaluation device is configured to evaluate the asset value of all objects based on the area value of each object and a preset asset value.
[0056] According to a tenth aspect of the embodiments of this specification, a computing device is provided, including:
[0057] memory and processor;
[0058] The memory is configured to store computer-executable instructions, and the processor is configured to execute the computer-executable instructions:
[0059] Obtaining a video to be processed, and segmenting the video to be processed to obtain one or more original video frames;
[0060] Obtaining target bounding boxes of all objects in the one or more original video frames, and determining motion trajectories of all objects based on the target bounding boxes;
[0061] Obtaining a target bounding box sequence corresponding to the motion trajectories of all objects, and determining an area value of each object in the video to be processed based on the target bounding box sequence;
[0062] The asset value of all objects is evaluated based on the area value of each object and the preset asset value.
[0063] According to an eleventh aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer instructions, and when the instructions are executed by a processor, the steps of the asset evaluation method are implemented.
[0064] One embodiment of the present specification implements a data processing method and device, and an image processing method and device, wherein the image processing method includes obtaining a video to be processed including a video frame sequence, and then obtaining a high-precision target bounding box for all objects in all video frames in the video frame sequence according to a preset detection method, and then determining the motion trajectory of each object based on the target bounding box to improve the robustness of motion trajectory tracking, and finally accurately determining the target object in the video according to the target bounding box sequence corresponding to the motion trajectory of each object. BRIEF DESCRIPTION OF THE DRAWINGS
[0065] Figure 1 This is an example diagram of an application scenario of a data processing method provided by an embodiment of this specification;
[0066] Figure 2 is a flow chart of a data processing method provided by one embodiment of this specification;
[0067] Figure 3 is a flowchart of an image processing method provided by one embodiment of this specification;
[0068] Figure 4 This is a schematic diagram of a specific process of obtaining an initial bounding box and a probability value of the initial bounding box in an image processing method provided by one embodiment of this specification;
[0069] Figure 5 This is a schematic diagram of a process for obtaining the motion trajectory of an object in an actual scene using an image processing method provided by an embodiment of this specification;
[0070] Figure 6 This is an embodiment of the present specification. Figure 5 The residual graph in the video frame realizes the process of obtaining the motion trajectory of the next object in the video frame;
[0071] Figure 7 This is a schematic diagram of the motion trajectories of all objects in a video frame obtained in an actual scene provided by an embodiment of this specification;
[0072] Figure 8 This is a structural diagram of a data processing device provided by one embodiment of this specification;
[0073] Figure 9This is a structural diagram of an image processing device provided by one embodiment of this specification;
[0074] Figure 10 This is a structural block diagram of a first computing device provided by an embodiment of this specification;
[0075] Figure 11 is a structural block diagram of a second computing device provided by an embodiment of this specification;
[0076] Figure 12 This is a flow chart of an asset evaluation method provided by one embodiment of this specification;
[0077] Figure 13 This is a schematic diagram of the structure of an asset evaluation device provided by one embodiment of this specification;
[0078] Figure 14 This is a structural block diagram of a third computing device provided by an embodiment of this specification. DETAILED DESCRIPTION
[0079] The following description sets forth many specific details to facilitate a thorough understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the scope of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.
[0080] The terms used in one or more embodiments of this specification are for the purpose of describing specific embodiments only and are not intended to limit one or more embodiments of this specification. The singular forms "a," "the," and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.
[0081] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".
[0082] First, the terms involved in one or more embodiments of this specification are explained.
[0083] Video clip: A video clip of a certain duration that can be broken down into consecutive video frames (i.e., images).
[0084] Person bounding box: A box used to mark the spatial area of a person in each video frame.
[0085] Person detector: Input a video frame / image and output the bounding box of the person in the video frame / image.
[0086] Person tracker: Input the bounding boxes of all video frames in the video and find the bounding box sequence of each person.
[0087] Subject locator: Find the corresponding action subject from the external frame sequence corresponding to each person in the video.
[0088] Fully convolutional network: A neural network with convolutional layers as output.
[0089] Bounding box rationality: An estimate that evaluates the rationality of a bounding box as a target bounding box.
[0090] Bounding box confidence: An estimate that evaluates the probability that a bounding box is credible as the target bounding box.
[0091] This specification provides a data processing method and an image processing method. One or more embodiments of this specification also involve a data processing device, an image processing device, an asset valuation method, an asset valuation device, three computing devices, and two computer-readable storage media, each of which is described in detail in the following embodiments.
[0092] See also Figure 1 and Figure 2 , Figure 1 An example diagram showing an application scenario of a data processing method provided according to an embodiment of this specification is shown. Figure 2 A flow chart of a data processing method provided according to an embodiment of the present specification is shown.
[0093] See also Figure 1 The application scenario includes a terminal and a server. Specifically, the user sends a video to be processed to the server through the terminal. After receiving the video to be processed, the server divides the video to be processed into one or more original video frames through a video splitter, obtains the target external bounding boxes of all objects in the one or more original video frames, and determines the motion trajectories of all objects based on the target external bounding boxes. Finally, it obtains the target external bounding box sequence corresponding to the motion trajectories of all objects, and determines the target objects in the video based on the target external bounding box sequence. The specific processing process can be seen in Figure 2 The following steps:
[0094] Step 202: Obtain the video to be processed.
[0095] During specific implementation, the video to be processed and the user's requirements for the video to be processed are obtained, wherein the processing requirement is to obtain the target object in the video to be processed. Specifically, if the video to be processed is a TV series, the target object can be the leading actor in the video to be processed.
[0096] Step 204: Segment the video to be processed to obtain one or more original video frames.
[0097] Step 206: Obtain target bounding boxes of all objects in the one or more original video frames, and determine motion trajectories of all objects based on the target bounding boxes.
[0098] Specifically, the target bounding boxes of all objects in the one or more original video frames are obtained through a human detector.
[0099] In addition, in actual applications, determining the motion trajectories of all objects based on the target bounding box specifically includes:
[0100] S1: put the target bounding box of each object in all video frames into the associated candidate set Objs;
[0101] S2: The target bounding box trajectory set Trks of all objects is set to an empty set, and the trajectory count k=0;
[0102] S3: Starting from the i-th frame of all video frames, determine the intersection-over-union ratio of the target bounding box of the j-th object in the i-th frame and the bounding boxes of all objects in the next frame, where i∈[1,n], and i and j are both positive integers;
[0103] S4: Determine whether the intersection-over-union ratio is greater than or equal to a preset threshold value T.
[0104] If yes, set target j as track k, add k to Trks, remove all target bounding boxes of the object corresponding to k from Objs, where target j is the target bounding box of the j-th object and the target bounding box with the largest intersection-over-union ratio of all object bounding boxes in the next frame, and execute step S5.
[0105] If not, increment i by 1 and execute step S3;
[0106] S5: Determine whether S1 is empty. If so, end; if not, continue executing S2.
[0107] Step 208: Obtain target bounding box sequences corresponding to the motion trajectories of all objects, and determine the target objects in the video based on the target bounding box sequences.
[0108] In a specific implementation, the areas of the target circumscribed frames in the target circumscribed frame sequence corresponding to the motion trajectories of all objects are added together, and the target object in the video is determined by the sum of the areas of the target circumscribed frames corresponding to the motion trajectories of all objects.
[0109] The data processing method provided in the embodiments of this specification includes dividing the acquired video into a video frame sequence, then obtaining a high-precision target bounding box for all objects in all video frames in the video frame sequence, and then determining the motion trajectory of each object based on the target bounding box to improve the robustness of motion trajectory tracking, and finally accurately determining the target object in the video based on the target bounding box sequence corresponding to the motion trajectory of each object.
[0110] See also Figure 3 , Figure 3 A flowchart of an image processing method provided according to an embodiment of the present specification is shown, including steps 302 to 312.
[0111] Step 302: Obtain a video to be processed, wherein the video includes a video frame sequence.
[0112] Step 304: Obtain all video frames of the video frame sequence, wherein the video frames include one or more objects.
[0113] The obtained video to be processed may be a video clip or a complete video set, and the video includes but is not limited to any type of video, such as self-media short videos, TV series, movies, news broadcasts, sports album programs, etc.
[0114] Specifically, the acquired video to be processed can be a video uploaded by a user or a video obtained through a secure interface of a third-party video platform. After the video to be processed is acquired, it is necessary to segment the video to be processed into a video frame sequence to obtain all video frames of the video frame sequence. In practical applications, any segmentation method that can segment a video into individual video frames can be used to segment the acquired video, and no limitation is imposed herein.
[0115] Step 306: Obtain the target bounding box of the one or more objects.
[0116] The object includes but is not limited to any object in the video. For example, if the video is a news video, the object can be all the people in the video, such as the host, interviewees, etc. If the video is an animation, the object can be all the animated characters in the video. If the video is a TV series, the object can be all the actors in the video.
[0117] For ease of understanding, the objects mentioned in this specification can be understood as real people. Specifically, the target bounding box of the one or more objects can be obtained, that is, the target bounding box of each person in all video frames of the video frame sequence can be obtained according to a preset detection method.
[0118] Specifically, the preset detection method includes but is not limited to any method that can obtain the target bounding box of each object in all video frames of the video frame sequence. For example, the preset detection method is to obtain the target person bounding box of all people in all video frames in the video frame sequence by using a person detector obtained by machine learning.
[0119] In specific implementation, the target bounding box of each object is obtained by fusing the initial bounding box and the probability value of the initial bounding box. The specific method is as follows:
[0120] The acquiring of the target bounding box of the one or more objects includes:
[0121] Obtaining an initial bounding box of each object in all video frames of the video frame sequence and a probability value of the initial bounding box according to a preset detection method;
[0122] When the probability value is greater than or equal to a preset probability threshold, the initial bounding box and the probability value of the initial bounding box are input into a fusion model for fusion to obtain a target bounding box for each object.
[0123] Specifically, first, the detector is used to obtain the initial bounding box of each object in each video frame and the probability value of each initial bounding box;
[0124] Then, a determination is made as to whether the probability value of each initial bounding box is greater than or equal to a preset probability threshold. If so, the initial bounding boxes with probability values greater than or equal to the preset probability threshold are fed into the fusion model with the corresponding probability values for fusion, obtaining the target bounding box of the object corresponding to the initial bounding box with probability values greater than or equal to the preset probability threshold. If not, the initial bounding box with probability values greater than or equal to the preset probability threshold and the object corresponding to the initial bounding box are deleted. The preset probability threshold can be set based on the actual application, for example, to 0.492 or 0.53. The greater the probability value of the initial bounding box is above the preset probability threshold, the greater the probability that the initial bounding box contains an object, that is, the greater the probability that it contains a person.
[0125] In practical applications, see Figure 4 The method of obtaining the initial bounding box of the object in all video frames of the video frame sequence and the probability value of the initial bounding box according to a preset detection method includes:
[0126] An initial bounding box of an object in all video frames of the video frame sequence and a probability value of the initial bounding box are obtained according to a convolutional neural network.
[0127] From the above, it can be seen that the preset detection method can be a human detector obtained through machine learning, and the human detector uses a CNN (Convolutional Neural Networks) model to obtain the initial bounding box of the object in all video frames and the probability value of the initial bounding box.
[0128] For a detailed process of obtaining the initial bounding boxes of objects in all video frames of the video frame sequence and the probability values of the initial bounding boxes by using the human detector, see Figure 4 , Figure 4 For each object's initial bounding box and a specific method for obtaining the probability value of the initial bounding box, first convert the video frame ( Figure 4 The image in the video frame is input into the CNN model to obtain the initial bounding box of each object in the video frame ( Figure 4 The person detection frame in the image), the rationality of each initial bounding box ( Figure 4 The rationality of the middle frame), the confidence of each initial bounding box ( Figure 4 Confidence of the middle box), then multiply the rationality of each initial bounding box by the confidence, and finally take the product of the rationality and confidence of each initial bounding box as the probability value of each initial bounding box ( Figure 4 The probability of the box in the middle is obtained by outputting the initial bounding box of each object and the probability value of each initial bounding box to achieve the subsequent acquisition of the target bounding box.
[0129] In addition, the fusion model includes but is not limited to the NMS (Non-maximum suppression) algorithm model. The main purpose of the NMS algorithm model is to eliminate redundant initial bounding boxes and find a better object detection position.
[0130] In an embodiment of the present specification, when acquiring the target bounding box of each object, the initial bounding box of each object in all video frames of the video frame sequence and the probability value of the initial bounding box are first obtained according to a preset detection method, and then the probability value is compared with the preset probability threshold to achieve a first screening of the initial bounding box. Finally, the initial bounding box with a probability value greater than or equal to the preset probability threshold and the probability value of the initial bounding box are input into the fusion model for fusion to achieve a second screening of the initial bounding box and obtain the target bounding box of each object. The target bounding box is obtained by the above two screening methods, so that the confidence of the obtained target bounding box is higher.
[0131] The probability value of the initial bounding box in the embodiment of this specification is obtained through the following steps:
[0132] The confidence and rationality of the initial bounding box of each object in all video frames of the video frame sequence are obtained according to a preset detection method.
[0133] The probability value of the initial bounding box is obtained by multiplying the confidence and rationality of the initial bounding box.
[0134] The confidence of the initial bounding box includes the probability that the object is contained in the initial bounding box, and the rationality of the initial bounding box includes the distance between the center of the initial bounding box and the center of the object.
[0135] For example, first, the confidence and rationality of the initial bounding box of each person in all video frames are obtained according to the person detector, and then the confidence and rationality of each person's initial bounding box are multiplied, and the product is used as the probability value of each person's initial bounding box, that is, the probability score of each initial bounding box.
[0136] The confidence of the initial bounding box is the classification score of the initial bounding box, which describes the probability that the initial bounding box contains a person; the rationality of the initial bounding box is the degree to which the center of the initial bounding box is close to the center of the person, which describes the accuracy of the initial bounding box.
[0137] In the embodiment of the present specification, the probability value of the initial bounding box is obtained by this multiplication method, and the target bounding box is obtained based on the initial bounding box and the probability value of the initial bounding box obtained in this way. This can greatly improve the confidence of the target bounding box and make the target bounding box closer to the included object, that is, it is more certain that the target bounding box contains a person, and the target bounding box is more closely aligned with the person.
[0138] Step 308: Determine the motion trajectory of the one or more objects based on the target bounding box.
[0139] Step 310: Obtain a target bounding box sequence corresponding to the motion trajectory of the one or more objects.
[0140] Specifically, determining the motion trajectory of the one or more objects based on the target bounding box includes:
[0141] The motion trajectory of each object is determined based on the association relationship between the target bounding box of each object and the previous and next video frames in all video frames of the video frame sequence.
[0142] For example, the motion trajectory of each object is determined based on the association relationship between the target bounding box of each object in the current video frame of the video frame sequence and the previous and next video frames.
[0143] The current frame is the current video frame, and the next frame is the next video frame. The next frame of the current frame can be the frame immediately adjacent to the current frame or the frame with an interval between the current frame and the current frame. For example, if video A is divided into video frame 1, video frame 2, and video frame 3, if video frame 1 is the current frame, then video frame 2 and video frame 3 can both be the next frames of video frame 1.
[0144] In a specific implementation, determining the motion trajectory of each object based on the association relationship between the target bounding box of each object and the previous and next video frames in all video frames of the video frame sequence specifically includes the following steps:
[0145] S1: Starting from the current frame, determine the target bounding box of the i-th object in the current frame and the target bounding box with the i-th object in the next frame, wherein the current frame is any frame in the video frame sequence, i∈[1,n], and i is a positive integer;
[0146] S2: determining a motion trajectory of the i-th object based on an association between a target bounding box of the i-th object in the current frame and a target bounding box having the i-th object in the next frame;
[0147] S3: Determine whether i is greater than n,
[0148] If so, then end.
[0149] If not, i is incremented by 1 and step S1 is executed.
[0150] Continuing with the above example, video A is segmented into video frames 1 to 3, and the current video frame is video frame 1.
[0151] Step 1: Determine the target bounding box of the first object a in video frame 1 and the target bounding box of object a in the next video frame 2 having object a;
[0152] Step 2: Determine the motion trajectory of object a based on the association between the target bounding box of the first object a in video frame 1 and the target bounding box of object a in video frame 2;
[0153] Step 3: Determine whether object a is the last object in video frame 1. If so, end the process and then determine the motion trajectory of each object in video frame 2 using the above steps. If not, continue to determine the motion trajectory of the next object after object a in video frame 1 using the above method.
[0154] In practical applications, the target bounding box of the i-th object in the current frame must be determined in the next frame before the motion trajectory of the i-th object in the current frame can be obtained. For example, to first determine whether object a exists in video frame 2, the specific processing method is as follows:
[0155] The determining of the target bounding box having the i-th object in the next frame includes:
[0156] Determine an intersection-over-union ratio of a target bounding box of the i-th object in the current frame and a target bounding box of each object in the next frame;
[0157] Determine whether there is an intersection-over-union ratio greater than or equal to a preset threshold,
[0158] If so, a target bounding box having the i-th object in the next frame is determined based on the intersection-over-union ratio.
[0159] Among them, the preset threshold can be set according to actual needs and is not limited here. Specifically, if the intersection and union ratio of the target bounding box of the i-th object and the target bounding box of each object in the next frame is greater than or equal to the preset threshold, it means that there is a target bounding box containing the i-th object in the next frame; if not, it means that there is no target bounding box containing the i-th object in the next frame. At this time, the motion trajectory of the i-th object can be determined based on the intersection and union ratio of the target bounding box of the i-th object in the current frame and the target bounding box of each object in the next frame. The specific processing process remains as described above.
[0160] In addition, when determining the next frame of the current frame, it is necessary to first determine whether the object in the current frame exists in the next frame to determine the next frame. Taking the above example, it is necessary to first determine whether video frame 2 contains the i-th object a in the current frame. If so, video frame 2 is used as the next frame.
[0161] If not, determine whether video frame 3 contains the i-th object a in the current frame;
[0162] If yes, take video frame 3 as the next frame;
[0163] If not, then end.
[0164] In the embodiment of this specification, when obtaining the motion trajectory of each object, it is obtained by matching the target bounding box of each object in the first segmented video frame with the target bounding box of each object in each video frame after the first video frame.
[0165] In the specific implementation, in order to prevent too many motion trajectories from affecting the acquisition of the motion trajectory of the next object, after each motion trajectory of an object is determined, the motion trajectory of the object will be recorded and deleted, as described below:
[0166] After determining the motion trajectory of the i-th object based on the association relationship between the target bounding box of the i-th object in the current frame and the target bounding box of the i-th object in the next frame, the method further includes:
[0167] Delete the motion trajectory of the i-th object.
[0168] See also Figure 5-Figure 7 , Figures 5 to 7 The human tracker uses a global graph greedy algorithm to find the optimal solution for tracking the motion trajectory of each object in all video frames.
[0169] See also Figure 5 , Figure 5 The figure is a schematic diagram of the specific processing process of determining the motion trajectory of an object in a video frame in an actual application scenario.
[0170] Step 1: Create a graph containing all data association hypotheses.
[0171] Specifically, taking the video being divided into three video frames: video frame A, video frame B, and video frame C as an example, each video frame includes three objects, wherein video frame A includes object a1, object a2, and object a3, video frame B includes object b1, object b2, and object b3, and video frame C includes object c1, object c2, and object c3.
[0172] First, establish the association relationships between objects a1, a2, and a3 in video frame A and objects b1, b2, and b3 in video frame B, respectively, and the association relationships between objects b1, b2, and b3 in video frame B and objects c1, c2, and c3 in video frame C, respectively, to create an object association relationship graph; that is, each object in each video frame is connected to all objects in the next video frame, so that each object in each video frame forms three edges with the objects in the next video frame.
[0173] Step 2: Find the resulting trajectory among all current data association hypotheses.
[0174] Specifically, taking object a1 in video frame A as an example, first determine the intersection-and-union ratios of the target bounding box of object a1 in video frame A with the target bounding boxes of objects b1, b2, and b3 in video frame B, respectively, and determine that the intersection-and-union ratio of the target bounding box of object a1 in video frame A and the target bounding box of object b2 in video frame B is the largest. Then, it can be determined that the target bounding box of object b2 in video frame B is the target bounding box with object a1 in video frame A, that is, object a1 is object b2 in video frame B, and then determine object c3 of object a1 in video frame C in the above manner. Finally, the edge from object a1 to object b2 and the edge from object b2 to object c3 are used as the result motion trajectory of object a1, and the result motion trajectory of each object is the motion trajectory that represents each object.
[0175] Step 3: Extract the result trajectory and store it in the solution space.
[0176] Specifically, the edge from object a1 to object b2 and the edge from object b2 to object c3 are extracted as the result motion trajectory of object a1 and stored in the solution space, where the solution space refers to a vector space formed by the set of all solutions of the homogeneous linear equations, that is, a set.
[0177] Step 4: Remove the last result trajectory from the data association hypothesis graph to obtain the residual graph.
[0178] Specifically, in order to obtain the result motion trajectory of object a2 in video A more clearly and accurately, the result motion trajectory of object a1 is deleted in the object association relationship diagram, and a residual graph after excluding the result motion trajectory of object a1 is obtained, so as to avoid repeated operations when extracting the result motion trajectory of a2, and quickly and accurately complete the acquisition of the result motion trajectories of other objects.
[0179] See also Figure 6 , Figure 6 Based on Figure 5 The residual map in is used to obtain the resulting motion trajectory of the next object in the video frame.
[0180] Still taking the above example, Figure 5 Repeat steps 1 to 4 above to obtain the result motion trajectory of object a3 [object a3, object b1, object c2] and a new residual map until the result motion trajectories of all objects are obtained.
[0181] See also Figure 7 , Figure 7 are the motion trajectories of all objects in the video frame, namely, motion trajectory track1 [object a1, object b2, object c3], motion trajectory track2 [object a3, object b1, object c2], and motion trajectory track3 [object a2, object b3, object c1].
[0182] In the embodiments of this specification, Figures 5 to 7 In practical applications, the tracker uses a global graph-based greedy algorithm to sequentially find the optimal solution for trajectory tracking. All object detections are represented as nodes, and all possible trajectory detections between different video frames are connected by edges, forming an association hypothesis graph for all objects. Then, based on the association hypothesis graph for all objects, the association between a certain object and the object in the next video frame is found. This is used as the resulting motion trajectory of the object currently extracted. This trajectory in the association hypothesis graph is stored and deleted, forming a residual graph for the next iteration and trajectory search for other objects. Steps 1 to 4 are repeated in each iteration to find the resulting motion trajectory and residual graph for that iteration. After all iterations are completed, the trajectories of all objects are collected. The advantages of this method for obtaining object motion trajectories are: (at least based on the global graph-based greedy algorithm) it can always find the globally optimal motion trajectory, improving robustness; and it does not require additional parameter settings, such as the maximum number of trajectories, making it easy to generalize.
[0183] Step 312: Determine the target object in the video based on the target bounding box sequence.
[0184] In another embodiment of the present specification, determining the target object in the video based on the target circumscribed frame sequence includes:
[0185] Determine the sum of the areas of all target bounding boxes in the target bounding box sequence corresponding to the motion trajectory of each object;
[0186] The motion trajectories of all objects are sorted based on the sum of the areas of all target bounding boxes, and the target object in the video is determined according to the sorting result.
[0187] In a specific implementation, determining the target object in the video according to the sorting result includes:
[0188] The highest-ranked motion track is determined according to the ranking result, and the object corresponding to the highest-ranked motion track is determined as the target object in the video.
[0189] In actual use, the category of the video to be processed can also be determined based on the objects in the video. For example, if the area occupied by the male and female protagonists among the objects in the video is larger, then the type of the video can be considered to be an emotional type. If the area occupied by weapons and the like among the objects in the video is larger, then the type of the video can be considered to be a war type, etc. The specific video type also needs to be determined based on the detected objects in the video and the area of the objects, and no limitation is made here.
[0190] In an embodiment of the present specification, after obtaining the motion trajectory of each object, a target bounding box sequence corresponding to the motion trajectory of each object is determined, and then the areas of the target bounding boxes in the target bounding box sequence are added together. Then, the object corresponding to the target bounding box sequence with the largest sum of the areas of the target bounding boxes in the target bounding box sequence is taken as the target object; the target object is more simply determined by summing the areas without the need for a special algorithm, making the processing process of the image processing method more convenient and quicker, which can greatly improve work efficiency.
[0191] In actual applications, the protagonist in a video generally occupies more video frames and close-up shots. It is more reasonable and convenient to determine the target object by the sum of the areas of the target external frames in the object's motion trajectory. Therefore, the method of determining the target object in this manual is more reliable and accurate.
[0192] Corresponding to the above method embodiment, this specification also provides a data processing device embodiment, Figure 8 FIG1 shows a schematic diagram of the structure of a data processing device provided by an embodiment of this specification. Figure 8 As shown, the device includes:
[0193] The demand acquisition module 802 is configured to acquire the video to be processed;
[0194] The video segmentation module 804 is configured to segment the video to be processed to obtain one or more original video frames;
[0195] The motion trajectory acquisition module 806 is configured to acquire target bounding boxes of all objects in the one or more original video frames, and determine the motion trajectories of all objects based on the target bounding boxes;
[0196] The object acquisition module 808 is configured to acquire a target bounding box sequence corresponding to the motion trajectories of all objects, and determine a target object in the video based on the target bounding box sequence.
[0197] The above is a schematic diagram of a data processing device according to this embodiment. It should be noted that the technical solution of the data processing device and the technical solution of the above-mentioned data processing method are based on the same concept. For details not described in detail in the technical solution of the data processing device, please refer to the description of the technical solution of the above-mentioned data processing method.
[0198] Corresponding to the above method embodiment, this specification also provides an image processing device embodiment, Figure 9 FIG. 1 shows a schematic diagram of the structure of an image processing device provided by an embodiment of this specification. Figure 9 As shown, the device includes:
[0199] The video acquisition module 902 is configured to acquire a video to be processed, wherein the video includes a video frame sequence;
[0200] The video frame obtaining module 904 is configured to obtain all video frames of the video frame sequence, wherein the video frames include one or more objects;
[0201] The bounding box obtaining module 906 is configured to obtain a target bounding box of the one or more objects;
[0202] a trajectory determination module 908 , configured to determine the motion trajectory of the one or more objects based on the target bounding box;
[0203] The external frame sequence acquisition module 910 is configured to acquire a target external frame sequence corresponding to the motion trajectory of the one or more objects;
[0204] The target object determination module 912 is configured to determine the target object in the video based on the target circumscribed frame sequence.
[0205] Optionally, the external frame acquisition module 910 is further configured to:
[0206] Obtaining an initial bounding box of each object in all video frames of the video frame sequence and a probability value of the initial bounding box according to a preset detection method;
[0207] When the probability value is greater than or equal to a preset probability threshold, the initial bounding box and the probability value of the initial bounding box are input into a fusion model for fusion to obtain a target bounding box for each object.
[0208] Optionally, the probability value of the initial bounding box is obtained by the following steps:
[0209] The confidence and rationality of the initial bounding box of each object in all video frames of the video frame sequence are obtained according to a preset detection method.
[0210] The probability value of the initial bounding box is obtained by multiplying the confidence and rationality of the initial bounding box.
[0211] The confidence of the initial bounding box includes the probability that the object is contained in the initial bounding box, and the rationality of the initial bounding box includes the degree of proximity between the center of the initial bounding box and the center of the object.
[0212] Optionally, the trajectory determination module 908 is further configured to:
[0213] The motion trajectory of each object is determined based on the association relationship between the target bounding box of each object and the previous and next video frames in all video frames of the video frame sequence.
[0214] Optionally, the trajectory determination module 908 is further configured to:
[0215] S1: Starting from the current frame, determine the target bounding box of the i-th object in the current frame and the target bounding box with the i-th object in the next frame, wherein the current frame is any frame in the video frame sequence, i∈[1,n], and i is a positive integer;
[0216] S2: determining a motion trajectory of the i-th object based on an association between a target bounding box of the i-th object in the current frame and a target bounding box having the i-th object in the next frame;
[0217] S3: Determine whether i is greater than n,
[0218] If so, then end.
[0219] If not, i is incremented by 1 and step S1 is executed.
[0220] Optionally, the trajectory determination module 908 is further configured to:
[0221] Determine an intersection-over-union ratio of a target bounding box of the i-th object in the current frame and a target bounding box of each object in the next frame;
[0222] Determine whether there is an intersection-over-union ratio greater than or equal to a preset threshold,
[0223] If so, a target bounding box having the i-th object in the next frame is determined based on the intersection-over-union ratio.
[0224] Optionally, the device further includes:
[0225] The deletion module is configured to delete the motion trajectory of the i-th object.
[0226] Optionally, the target object determination module 912 is further configured to:
[0227] Determine the sum of the areas of all target bounding boxes in the target bounding box sequence corresponding to the motion trajectory of each object;
[0228] The motion trajectories of all objects are sorted based on the sum of the areas of all target bounding boxes, and the target object in the video is determined according to the sorting result.
[0229] Optionally, the target object determination module 912 is further configured to:
[0230] The highest-ranked motion track is determined according to the ranking result, and the object corresponding to the highest-ranked motion track is determined as the target object in the video.
[0231] Optionally, the external frame acquisition module 906 is further configured to:
[0232] An initial bounding box of an object in all video frames of the video frame sequence and a probability value of the initial bounding box are obtained according to a convolutional neural network.
[0233] The image processing device provided in the embodiments of this specification includes dividing the acquired video into a video frame sequence, and then obtaining a high-precision target bounding box for all objects in all video frames in the video frame sequence according to a preset detection method, and then determining the motion trajectory of each object based on the target bounding box to improve the robustness of motion trajectory tracking, and finally accurately determining the target object in the video according to the target bounding box sequence corresponding to the motion trajectory of each object.
[0234] The above is a schematic diagram of an image processing device according to this embodiment. It should be noted that the technical solution of the image processing device and the technical solution of the above-mentioned image processing method are based on the same concept. For details not described in detail in the technical solution of the image processing device, please refer to the description of the technical solution of the above-mentioned image processing method.
[0235] Figure 10 The block diagram of the first computing device 1000 according to one embodiment of the present disclosure is shown. Components of the computing device 1000 include, but are not limited to, a memory 1010 and a processor 1020. The processor 1020 is connected to the memory 1010 via a bus 1030, and a database 1050 is used to store data.
[0236] The computing device 1000 also includes an access device 1040 that enables the computing device 1000 to communicate via one or more networks 1060. Examples of such networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 1040 may include one or more of any type of network interface (e.g., a network interface card (NIC)) whether wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.
[0237] In one embodiment of the present specification, the above components of the computing device 1000 and Figure 10Other components not shown in the figure may also be connected to each other, for example, via a bus. Figure 10 The computing device structure block diagram shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art may add or replace other components as needed.
[0238] Computing device 1000 can be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, personal digital assistant, laptop computer, notebook computer, netbook computer, etc.), a mobile phone (e.g., a smartphone), a wearable computing device (e.g., a smartwatch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or PC. Computing device 1000 can also be a mobile or stationary server.
[0239] The processor 1020 is configured to execute the following computer-executable instructions:
[0240] Get the video to be processed;
[0241] Segmenting the video to be processed to obtain one or more original video frames;
[0242] Obtaining target bounding boxes of all objects in the one or more original video frames, and determining motion trajectories of all objects based on the target bounding boxes;
[0243] A target bounding box sequence corresponding to the motion trajectories of all objects is obtained, and a target object in the video is determined based on the target bounding box sequence.
[0244] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the above-mentioned data processing method are based on the same concept. For details not described in detail in the technical scheme of the computing device, please refer to the description of the technical scheme of the above-mentioned data processing method.
[0245] Figure 11 The following is a block diagram of a second computing device 1100 according to one embodiment of the present disclosure. The components of the computing device 1100 include, but are not limited to, a memory 1110 and a processor 1120. The processor 1120 is connected to the memory 1110 via a bus 1130, and a database 1150 is used to store data.
[0246] The computing device 1100 also includes an access device 1140 that enables the computing device 1100 to communicate via one or more networks 1060. Examples of such networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 1140 may include one or more of any type of network interface (e.g., a network interface card (NIC)) whether wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.
[0247] In one embodiment of the present specification, the above components of the computing device 1100 and Figure 11 Other components not shown in the figure may also be connected to each other, for example, via a bus. Figure 11 The computing device structure block diagram shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art may add or replace other components as needed.
[0248] Computing device 1100 can be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, personal digital assistant, laptop computer, notebook computer, netbook computer, etc.), a mobile phone (e.g., a smartphone), a wearable computing device (e.g., a smartwatch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or PC. Computing device 1100 can also be a mobile or stationary server.
[0249] The processor 1120 is configured to execute the following computer-executable instructions:
[0250] Acquire a video to be processed, wherein the video includes a video frame sequence;
[0251] Obtaining all video frames of the video frame sequence, wherein the video frames include one or more objects;
[0252] Obtaining a target bounding box of the one or more objects;
[0253] determining a motion trajectory of the one or more objects based on the target bounding box;
[0254] Obtaining a target bounding box sequence corresponding to the motion trajectory of the one or more objects;
[0255] A target object in the video is determined based on the target bounding box sequence.
[0256] The above is a schematic diagram of a computing device according to this embodiment. It should be noted that the technical solution of the computing device and the technical solution of the above-mentioned image processing method are based on the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the above-mentioned image processing method.
[0257] An embodiment of the present specification further provides a computer-readable storage medium storing computer instructions, which are used to implement the steps of the data processing method or the image processing method when executed by a processor.
[0258] The above is a schematic diagram of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium is based on the same concept as the technical solution of the aforementioned data processing method or image processing method. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the aforementioned data processing method or image processing method.
[0259] See also Figure 12 , Figure 12 A flowchart of an asset evaluation method provided according to an embodiment of this specification is shown, including steps 1202 to 1208.
[0260] Step 1202: Obtain a video to be processed, and segment the video to be processed to obtain one or more original video frames.
[0261] Step 1204: Obtain target bounding boxes of all objects in the one or more original video frames, and determine motion trajectories of all objects based on the target bounding boxes.
[0262] Step 1206: Obtain a target bounding box sequence corresponding to the motion trajectories of all objects, and determine the area value of each object in the video to be processed based on the target bounding box sequence.
[0263] The specific processing methods of steps 1202 to 1206 can be found in the above-mentioned embodiment of the image processing method, and will not be described in detail here.
[0264] Specifically, determining the area value of each object in the video to be processed based on the target bounding box sequence includes:
[0265] The sum of the areas of all target bounding boxes in the target bounding box sequence corresponding to the motion trajectory of each object is determined, where the sum of the areas represents the area value of each object in the video to be processed.
[0266] For example, the target bounding box sequence corresponding to the motion trajectory of object c is [c1, c2], then the area value of object c in the video to be processed is the sum of the areas of the target bounding boxes of c1 and c2.
[0267] Step 1208: Evaluate the asset value of all objects based on the area value of each object and the preset asset value.
[0268] In a specific implementation, the asset value evaluation of all objects based on the area value of each object and the preset asset value includes:
[0269] Sort all objects based on the area value of each object to obtain a first sorting result;
[0270] sorting all objects based on a preset asset value of each object to obtain a second sorting result;
[0271] Determine whether the first sorting result matches the second sorting result,
[0272] If so, the asset value of the object is determined to be reasonable.
[0273] In addition, if the first sorting result does not match the second sorting result, it means that the asset value of the object is unreasonable, and the asset value of the object needs to be re-determined based on the correlation between the area value and the asset value in the balance sheet in the asset database.
[0274] In practical applications, the asset value of an object can be understood as the value of the object. For example, if the object is an actor in a video, the asset value of the object can be the fee paid by the video producer to the actor, that is, the value of the actor in the video.
[0275] In actual applications, the preset asset value is the actual asset value of the object, that is, the actual value. In specific implementation, the larger the area occupied by the object in the video, the higher its corresponding asset value should be. If the area value of the object is large and the actual asset value is low, then the asset value of the object is unreasonable and may need to be readjusted.
[0276] For example, the video to be processed includes objects 1, 2, and 3, where the area of object 1 in the video to be processed is 30m. 2 , the area value of object 2 in the video to be processed is 20m 2 , the area value of object 3 in the video to be processed is 50m 2, and the actual asset value of object 1 is 20W, the actual asset value of object 2 is 30W, and the actual asset value of object 3 is 25W. It can be clearly seen that object 3 has the largest area value in the video to be processed, but the actual asset value of object 3 is not the highest. Object 2 has the smallest area value in the video to be processed, but the actual asset value of object 2 is the highest. At this time, it can be determined that the asset value distribution of objects 2 and 3 is unreasonable and needs to be readjusted.
[0277] In a specific implementation, the target asset value of each object may be determined based on the matching relationship between the area value of each object in the video to be processed and the asset value in the asset database.
[0278] Among them, the asset database includes an asset table in which area values are correlated with asset values. Specifically, after obtaining the area value of each object in the video to be processed, the target asset value corresponding to the area value of each object in the video to be processed can be obtained based on the asset table.
[0279] In another embodiment of the present specification, if it is only necessary to obtain the asset value corresponding to the object based on the area value of the object, the asset value of the object can be directly obtained based on the asset table, as follows:
[0280] After obtaining a target bounding box sequence corresponding to the motion trajectories of all objects and determining the area value of each object in the video to be processed based on the target bounding box sequence, the method further includes:
[0281] Based on a matching relationship between the area value of each object in the video to be processed and the asset value in the asset database, a target asset value of each object is determined.
[0282] For example, if the object is an advertisement, an advertising fee for the advertisement may be charged based on the area of the advertisement in the video.
[0283] The asset valuation method provided in the embodiments of this specification can evaluate whether the asset value of an object is reasonable based on the area value occupied by the object in the video to be processed. It can play an audit role in the assets of the object, avoid unreasonable phenomena, and provide a good user experience. It can also directly determine the actual asset value of the object based on the area value occupied by the object in the video to be processed, and the asset determination is convenient and fast.
[0284] See also Figure 13 , Figure 13 An asset evaluation device according to one embodiment of this specification is shown, comprising:
[0285] The video frame obtaining device 1302 is configured to obtain a video to be processed and segment the video to be processed to obtain one or more original video frames;
[0286] The trajectory determination device 1304 is configured to obtain target bounding boxes of all objects in the one or more original video frames, and determine the motion trajectories of all objects based on the target bounding boxes;
[0287] The area value determining means 1306 is configured to obtain a target bounding box sequence corresponding to the motion trajectories of all objects, and determine an area value of each object in the video to be processed based on the target bounding box sequence;
[0288] The evaluation device 1308 is configured to evaluate the asset value of all objects based on the area value of each object and the preset asset value.
[0289] Optionally, the area value determining means 1306 is further configured to:
[0290] The sum of the areas of all target bounding boxes in the target bounding box sequence corresponding to the motion trajectory of each object is determined, where the sum of the areas represents the area value of each object in the video to be processed.
[0291] Optionally, the evaluation device 1308 is further configured to:
[0292] Sort all objects based on the area value of each object to obtain a first sorting result;
[0293] sorting all objects based on a preset asset value of each object to obtain a second sorting result;
[0294] Determine whether the first sorting result matches the second sorting result,
[0295] If so, the asset value of the object is determined to be reasonable.
[0296] Optionally, the device further includes:
[0297] The target asset value determination module is configured to determine the target asset value of each object based on a matching relationship between the area value of each object in the video to be processed and the asset value in the asset database.
[0298] The above is a schematic diagram of an asset evaluation device according to this embodiment. It should be noted that the technical solution of the asset evaluation device and the technical solution of the aforementioned asset evaluation method are based on the same concept. For details not described in detail in the technical solution of the asset evaluation device, please refer to the description of the technical solution of the aforementioned asset evaluation method.
[0299] Figure 14The block diagram of a third computing device 1400 according to one embodiment of the present disclosure is shown. Components of the computing device 1400 include, but are not limited to, a memory 1410 and a processor 1420. The processor 1420 is connected to the memory 1410 via a bus 1430, and a database 1450 is used to store data.
[0300] Computing device 1400 also includes an access device 1440 that enables computing device 1400 to communicate via one or more networks 1460. Examples of such networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. Access device 1440 may include one or more of any type of network interface (e.g., a network interface card (NIC)), whether wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.
[0301] In one embodiment of the present specification, the above components of the computing device 1400 and Figure 14 Other components not shown in the figure may also be connected to each other, for example, via a bus. Figure 14 The computing device structure block diagram shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art may add or replace other components as needed.
[0302] Computing device 1400 can be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, personal digital assistant, laptop computer, notebook computer, netbook computer, etc.), a mobile phone (e.g., a smartphone), a wearable computing device (e.g., a smartwatch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or PC. Computing device 1400 can also be a mobile or stationary server.
[0303] The processor 1420 is configured to execute the following computer-executable instructions:
[0304] Obtaining a video to be processed, and segmenting the video to be processed to obtain one or more original video frames;
[0305] Obtaining target bounding boxes of all objects in the one or more original video frames, and determining motion trajectories of all objects based on the target bounding boxes;
[0306] Obtaining a target bounding box sequence corresponding to the motion trajectories of all objects, and determining an area value of each object in the video to be processed based on the target bounding box sequence;
[0307] The asset value of all objects is evaluated based on the area value of each object and the preset asset value.
[0308] The above is a schematic diagram of a computing device according to this embodiment. It should be noted that the technical solution of the computing device and the technical solution of the aforementioned asset valuation method are based on the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the aforementioned asset valuation method.
[0309] An embodiment of the present specification further provides a computer-readable storage medium storing computer instructions, which are used to implement the steps of the asset evaluation method when executed by a processor.
[0310] The above is a schematic diagram of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium is based on the same concept as the technical solution of the asset valuation method or image processing method described above. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the asset valuation method described above.
[0311] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0312] The computer instructions include computer program code, which may be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium may include any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal, and software distribution medium. It should be noted that the content of the computer-readable medium may be appropriately increased or decreased based on the requirements of legislation and patent practice within a jurisdiction. For example, in some jurisdictions, based on legislation and patent practice, computer-readable media does not include electric carrier signals and telecommunication signals.
[0313] It should be noted that for the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.
[0314] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0315] The preferred embodiments disclosed above are intended only to help illustrate this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific embodiments described. Obviously, many modifications and variations can be made based on the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that those skilled in the art can better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.
Claims
1. A data processing method, comprising: Get the video to be processed; Segmenting the video to be processed to obtain one or more original video frames; Obtaining target bounding boxes of all objects in the one or more original video frames, wherein the target bounding box of each object is obtained by fusing an initial bounding box and a probability value of the initial bounding box, wherein the probability value of the initial bounding box is the product of the confidence and rationality of the initial bounding box, and determining the motion trajectories of all objects based on the target bounding box, wherein determining the motion trajectories of all objects based on the target bounding box includes: determining the motion trajectories of all objects based on the association relationship between the target bounding box and the previous and next video frames; A target circumscribed frame sequence corresponding to the motion trajectories of all objects is obtained, and a target object in the video is determined based on the area of the target circumscribed frame sequence, and an object corresponding to a target circumscribed frame sequence having the largest sum of the areas of the target circumscribed frame sequences is determined as the target object.
2. An image processing method, comprising: Acquire a video to be processed, wherein the video includes a video frame sequence; Obtaining all video frames of the video frame sequence, wherein the video frames include one or more objects; Acquire a target bounding box of the one or more objects, wherein the target bounding box of each object is obtained by fusing an initial bounding box and a probability value of the initial bounding box, and the probability value of the initial bounding box is the product of a confidence level and a rationality level of the initial bounding box; Based on the target circumscribed frame, determining the motion trajectories of the one or more objects, wherein determining the motion trajectories of all objects based on the target circumscribed frame comprises: determining the motion trajectories of all objects based on the association relationship between the target circumscribed frame and previous and next video frames; Obtaining a target bounding box sequence corresponding to the motion trajectory of the one or more objects; A target object in the video is determined based on the areas of the target circumscribed frame sequence, and an object corresponding to a target circumscribed frame sequence having the largest sum of areas of the target circumscribed frame sequences is determined as the target object.
3. The image processing method according to claim 2, wherein obtaining a target bounding box of the one or more objects comprises: Obtaining an initial bounding box of each object in all video frames of the video frame sequence and a probability value of the initial bounding box according to a preset detection method; When the probability value is greater than or equal to a preset probability threshold, the initial bounding box and the probability value of the initial bounding box are input into a fusion model for fusion to obtain a target bounding box for each object.
4. The image processing method according to claim 3, wherein the probability value of the initial bounding box is obtained by the following steps: The confidence and rationality of the initial bounding box of each object in all video frames of the video frame sequence are obtained according to a preset detection method. And the probability value of the initial bounding box is obtained by multiplying the confidence and rationality of the initial bounding box. in, The confidence of the initial circumscribed frame includes the probability that the object is contained in the initial circumscribed frame, and the rationality of the initial circumscribed frame includes the degree of proximity between the center of the initial circumscribed frame and the center of the object.
5. The image processing method according to claim 2, wherein determining the motion trajectory of the one or more objects based on the target bounding box comprises: The motion trajectory of each object is determined based on the association relationship between the target bounding box of each object and the previous and next video frames in all video frames of the video frame sequence.
6. The image processing method according to claim 2, wherein determining the motion trajectory of each object based on the association relationship between the target bounding box of each object in all video frames of the video frame sequence and the preceding and following video frames comprises: S1: Starting from the current frame, determine the target bounding box of the i-th object in the current frame and the target bounding box with the i-th object in the next frame, wherein the current frame is any frame in the video frame sequence, i∈[1,n], and i is a positive integer; S2: determining a motion trajectory of the i-th object based on an association relationship between a target bounding box of the i-th object in the current frame and a target bounding box of the i-th object in the next frame; S3: Determine whether i is greater than n, If yes, then end. If not, i is incremented by 1 and step S1 is executed.
7. The image processing method according to claim 6, wherein determining the target bounding box having the i-th object in the next frame comprises: Determine an intersection-over-union ratio of a target bounding box of the i-th object in the current frame and a target bounding box of each object in the next frame; Determine whether there is an intersection-over-union ratio greater than or equal to a preset threshold, If so, a target bounding box having the i-th object in the next frame is determined based on the intersection-over-union ratio.
8. The image processing method according to claim 6, after determining the motion trajectory of the i-th object based on the association relationship between the target bounding box of the i-th object in the current frame and the target bounding box with the i-th object in the next frame, further comprising: Delete the motion track of the i-th object.
9. The image processing method according to claim 2, wherein determining the target object in the video based on the area of the target bounding box sequence comprises: Determine the sum of the areas of all target bounding boxes in the target bounding box sequence corresponding to the motion trajectory of each object; The motion trajectories of all objects are sorted based on the sum of the areas of all target bounding boxes, and the target object in the video is determined according to the sorting result.
10. The image processing method according to claim 9, wherein determining the target object in the video according to the sorting result comprises: The highest-ranked motion track is determined according to the ranking result, and the object corresponding to the highest-ranked motion track is determined as the target object in the video.
11. The image processing method according to claim 3, wherein obtaining the initial bounding box of the object in all video frames of the video frame sequence and the probability value of the initial bounding box according to a preset detection method comprises: An initial bounding box of an object in all video frames of the video frame sequence and a probability value of the initial bounding box are obtained according to a convolutional neural network.
12. A data processing device, comprising: A demand acquisition module is configured to acquire a video to be processed; A video segmentation module is configured to segment the video to be processed to obtain one or more original video frames; The motion trajectory acquisition module is configured to acquire the target bounding box of all objects in the one or more original video frames, wherein the target bounding box of each object is obtained by fusing the initial bounding box and the probability value of the initial bounding box, the probability value of the initial bounding box is the product of the confidence and rationality of the initial bounding box, and determine the motion trajectory of all objects based on the target bounding box, wherein the determining the motion trajectory of all objects based on the target bounding box includes: determining the motion trajectory of all objects based on the association relationship between the target bounding box and the previous and next video frames; The object acquisition module is configured to obtain the areas of the target circumscribed frame sequence corresponding to the motion trajectories of all objects, and determine the target object in the video based on the target circumscribed frame sequence, and determine the object corresponding to the target circumscribed frame sequence with the largest sum of the areas of the target circumscribed frame in the target circumscribed frame sequence as the target object.
13. An image processing device, comprising: A video acquisition module is configured to acquire a video to be processed, wherein the video includes a video frame sequence; A video frame acquisition module, configured to acquire all video frames of the video frame sequence, wherein the video frames include one or more objects; an external frame acquisition module, configured to acquire a target external frame of the one or more objects, wherein the target external frame of each object is obtained by fusing an initial external frame and a probability value of the initial external frame, and the probability value of the initial external frame is the product of a confidence level and a rationality level of the initial external frame; A trajectory determination module is configured to determine the motion trajectory of the one or more objects based on the target circumscribed frame, wherein the determining the motion trajectory of all objects based on the target circumscribed frame comprises: determining the motion trajectory of all objects based on the association relationship between the target circumscribed frame and the previous and next video frames; An external frame sequence acquisition module is configured to acquire a target external frame sequence corresponding to the motion trajectory of the one or more objects; The target object determination module is configured to determine the target object in the video based on the area of the target circumscribed frame sequence, and determine the object corresponding to the target circumscribed frame sequence with the largest sum of the areas of the target circumscribed frames in the target circumscribed frame sequence as the target object.
14. A computing device comprising: Memory and processor; The memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions: Get the video to be processed; Segmenting the video to be processed to obtain one or more original video frames; Obtaining target bounding boxes of all objects in the one or more original video frames, wherein the target bounding box of each object is obtained by fusing an initial bounding box and a probability value of the initial bounding box, wherein the probability value of the initial bounding box is the product of the confidence and rationality of the initial bounding box, and determining the motion trajectories of all objects based on the target bounding box, wherein determining the motion trajectories of all objects based on the target bounding box includes: determining the motion trajectories of all objects based on the association relationship between the target bounding box and the previous and next video frames; A target circumscribed frame sequence corresponding to the motion trajectories of all objects is obtained, and a target object in the video is determined based on the area of the target circumscribed frame sequence, and an object corresponding to a target circumscribed frame sequence having the largest sum of the areas of the target circumscribed frame sequences is determined as the target object.
15. A computing device comprising: Memory and processor; The memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions: Acquire a video to be processed, wherein the video includes a video frame sequence; Obtaining all video frames of the video frame sequence, wherein the video frames include one or more objects; Acquire a target bounding box of the one or more objects, wherein the target bounding box of each object is obtained by fusing an initial bounding box and a probability value of the initial bounding box, and the probability value of the initial bounding box is the product of a confidence level and a rationality level of the initial bounding box; Based on the target circumscribed frame, determining the motion trajectories of the one or more objects, wherein determining the motion trajectories of all objects based on the target circumscribed frame comprises: determining the motion trajectories of all objects based on the association relationship between the target circumscribed frame and previous and next video frames; Obtaining a target bounding box sequence corresponding to the motion trajectory of the one or more objects; A target object in the video is determined based on the areas of the target circumscribed frame sequence, and an object corresponding to a target circumscribed frame sequence having the largest sum of areas of the target circumscribed frame sequences is determined as the target object.
16. A computer-readable storage medium storing computer instructions, which, when executed by a processor, implement the data processing method of claim 1 or the steps of the image processing method of any one of claims 2 to 11.
17. An asset valuation method comprising: Acquire a video to be processed, and segment the video to be processed to obtain one or more original video frames; Obtaining target bounding boxes of all objects in the one or more original video frames, wherein the target bounding box of each object is obtained by fusing an initial bounding box and a probability value of the initial bounding box, wherein the probability value of the initial bounding box is the product of the confidence and rationality of the initial bounding box, and determining the motion trajectories of all objects based on the target bounding box, wherein determining the motion trajectories of all objects based on the target bounding box includes: determining the motion trajectories of all objects based on the association relationship between the target bounding box and the previous and next video frames; Obtaining a target bounding box sequence corresponding to the motion trajectories of all objects, and determining an area value of each object in the video to be processed based on the target bounding box sequence; The asset value of all objects is evaluated based on the area value of each object and the preset asset value.
18. The asset evaluation method according to claim 17, wherein determining the area value of each object in the video to be processed based on the target bounding box sequence comprises: The sum of the areas of all target bounding boxes in the target bounding box sequence corresponding to the motion trajectory of each object is determined, and the sum of the areas represents the area value of each object in the video to be processed.
19. The asset evaluation method according to claim 18, wherein the evaluation of the asset values of all objects based on the area value of each object and the preset asset value comprises: Sort all objects based on the area value of each object to obtain a first sorting result; sorting all objects based on a preset asset value of each object to obtain a second sorting result; Determine whether the first sorting result matches the second sorting result, If so, the asset value of the object is determined to be reasonable.
20. The asset evaluation method according to claim 17, after obtaining the target bounding box sequence corresponding to the motion trajectories of all objects and determining the area value of each object in the video to be processed based on the target bounding box sequence, further comprising: Based on the matching relationship between the area value of each object in the video to be processed and the asset value in the asset database, the target asset value of each object is determined.
21. An asset evaluation device, comprising: The video frame obtaining device is configured to obtain a video to be processed and segment the video to be processed to obtain one or more original video frames; The trajectory determination device is configured to obtain a target bounding box of all objects in the one or more original video frames, wherein the target bounding box of each object is obtained by fusing an initial bounding box and a probability value of the initial bounding box, wherein the probability value of the initial bounding box is the product of the confidence and rationality of the initial bounding box, and determine the motion trajectories of all objects based on the target bounding box, wherein determining the motion trajectories of all objects based on the target bounding box includes: determining the motion trajectories of all objects based on the association relationship between the target bounding box and the previous and next video frames; An area value determination device is configured to obtain a target circumscribed frame sequence corresponding to the motion trajectories of all objects, and determine the area value of each object in the video to be processed based on the target circumscribed frame sequence; The evaluation device is configured to evaluate the asset value of all objects based on the area value of each object and a preset asset value.
22. A computing device comprising: Memory and processor; The memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions: Acquire a video to be processed, and segment the video to be processed to obtain one or more original video frames; Obtaining target bounding boxes of all objects in the one or more original video frames, wherein the target bounding box of each object is obtained by fusing an initial bounding box and a probability value of the initial bounding box, wherein the probability value of the initial bounding box is the product of the confidence and rationality of the initial bounding box, and determining the motion trajectories of all objects based on the target bounding box, wherein determining the motion trajectories of all objects based on the target bounding box includes: determining the motion trajectories of all objects based on the association relationship between the target bounding box and the previous and next video frames; Obtaining a target bounding box sequence corresponding to the motion trajectories of all objects, and determining an area value of each object in the video to be processed based on the target bounding box sequence; The asset value of all objects is evaluated based on the area value of each object and the preset asset value.
23. A computer-readable storage medium storing computer instructions, which, when executed by a processor, implement the steps of the asset evaluation method described in any one of claims 17 to 20.
Citation Information
Patent Citations
Target detection-based tracking method and device, and terminal equipment
CN110047095A
