Method and device for classifying objects in video data, processor and electronic equipment

Abstract

The invention discloses a method and a device for classifying objects in video data, a processor and electronic equipment. The method comprises the steps that video data collected in a target scene are acquired, and the video data comprise at least one piece of video frame data; at least one initial object of video frame data in the video data is detected, at least one first detection result is obtained, and the first detection result is used for at least representing an initial category to which the initial object belongs; and based on the first detection result, determining a target category to which at least one target object of the video data belongs, the target object including the initial object. According to the invention, the technical problem of low classification precision of objects in video data is solved.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a method, device, processor and electronic device for classifying objects in video data. Background Art

[0002] In the process of processing video data, it often takes a lot of time to track the changing objects. Even if the change track of the object is obtained, it may not be possible to classify the object into the ideal category, which leads to the technical problem of low classification accuracy of the objects in the video data.

[0003] Currently, no effective solution has been proposed to the technical problem of low classification accuracy of objects in the above-mentioned video data. Summary of the invention

[0004] Embodiments of the present invention provide a method, device, processor, and electronic device for classifying objects in video data, so as to at least solve the technical problem of low classification accuracy of objects in video data.

[0005] According to one aspect of an embodiment of the present invention, a method for classifying objects in video data is provided, the method comprising: acquiring video data collected in a target scene, wherein the video data includes at least one video frame data; detecting at least one initial object of the video frame data in the video data to obtain at least one first detection result, wherein the first detection result is used to at least indicate an initial category to which the initial object belongs; based on the first detection result, determining a target category to which at least one target object of the video data belongs, wherein the target object includes the initial object.

[0006] Optionally, at least one initial object in the video frame data in the video data is detected to obtain at least one first detection result, including: generating at least one candidate frame in the video frame data, wherein the candidate frame is used to select the initial object; and detecting at least one initial object in multiple candidate frames to obtain at least one first detection result.

[0007] Optionally, in multiple candidate frames, at least one initial object is detected to obtain at least one first detection result, including: in response to the multiple candidate frames including candidate frames corresponding to different categories, the multiple candidate frames are merged, wherein the overlap between the candidate frames corresponding to different categories exceeds the target overlap; in the merged candidate frames, at least one initial object is detected to obtain at least one first detection result.

[0008] Optionally, based on the first detection result, determining the target category to which at least one target object of the video data belongs includes: determining an association relationship between a first initial object and a second initial object from the first detection result, wherein the at least one initial object includes the first initial object and the second initial object, and a first position of the first initial object in the target scene is different from a second position of the second initial object in the target scene; in response to the association relationship indicating that the first initial object is attached to the second initial object, determining a current candidate feature of the second initial object in the video frame data; and determining the target category based on the current candidate feature.

[0009] Optionally, based on the current candidate feature, determining the target category includes: determining the previous candidate feature that the second initial object has in the previous video frame data of the video frame data; matching the current candidate feature with the previous candidate feature to obtain a matching result; determining matching information between the matching result and the motion state of the second initial object, wherein the matching information is used to represent the matching cost between the matching result and the motion state; and determining the target category based on the matching information.

[0010] Optionally, determining the target category based on the matching information includes: in response to the matching information satisfying the target matching information and the tracking of the second initial object has ended, detecting the second initial object in the previous video frame data to obtain a second detection result, wherein the second detection result is used to indicate the initial category to which the second initial object belongs; determining the target category based on the first detection result and the second detection result.

[0011] According to one aspect of an embodiment of the present invention, a device for classifying objects in video data is provided, and the device may include: an acquisition unit, used to acquire video data collected in a target scene, wherein the video data includes at least one video frame data; a detection unit, used to detect at least one initial object of the video frame data in the video data to obtain at least one first detection result, wherein the first detection result is used to at least indicate an initial category to which the initial object belongs; a determination unit, used to determine, based on the first detection result, a target category to which at least one target object of the video data belongs, wherein the target object includes the initial object.

[0012] According to another aspect of an embodiment of the present invention, a processor is further provided, wherein the processor is used to run a program, wherein when the program is run by the processor, the method for classifying objects in video data in an embodiment of the present invention is executed.

[0013] According to another aspect of an embodiment of the present invention, an electronic device is provided, including: a memory storing an executable program; and a processor for running the program, wherein when the program is running, the method for classifying objects in video data in various embodiments of the present invention is executed.

[0014] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method for classifying objects in video data in the present invention.

[0015] According to another aspect of an embodiment of the present invention, a computer program product is further provided. The computer program product includes a computer program. When the computer program is executed by a processor, the method for classifying objects in video data in an embodiment of the present invention is implemented.

[0016] According to another aspect of an embodiment of the present invention, a computer program product is also provided, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium is used to store a computer program, and when the computer program is executed by a processor, the method for classifying objects in video data in an embodiment of the present invention is implemented.

[0017] According to another aspect of the embodiments of the present invention, the embodiments of the present application further provide a computer program, which, when executed by a processor, implements the method for classifying objects in video data in the above-mentioned embodiments of the present invention.

[0018] In an embodiment of the present invention, when classifying objects in video data, video data collected in a target scene can be obtained. At least one initial object of the video frame data in the acquired video data is detected to obtain at least one first detection result, and according to the obtained first detection result, the target category to which at least one target object of the video data belongs can be determined, thereby achieving the purpose of real-time tracking of objects in change, thereby solving the technical problem of low classification accuracy of objects in video data, and further achieving the technical effect of improving the classification accuracy of objects in video data. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0020] Figure 1 is a flow chart of a method for classifying objects in video data according to an embodiment of the present invention;

[0021] FIG2( a ) is a flow chart of a video parsing method according to an embodiment of the present invention;

[0022] FIG2( b ) is a schematic diagram of a feature matching effect according to an embodiment of the present invention;

[0023] FIG2( c ) is a schematic diagram of an affiliation relationship according to an embodiment of the present invention;

[0024] FIG2( d ) is a schematic diagram of an object tracking architecture according to an embodiment of the present invention;

[0025] FIG2( e ) is a schematic diagram of a label fusion effect according to an embodiment of the present invention;

[0026] Figure 3 is a schematic diagram of a device for classifying objects in video data according to an embodiment of the present invention. DETAILED DESCRIPTION

[0027] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only embodiments of a part of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0028] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0029] According to an embodiment of the present invention, a method for classifying objects in video data is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0030] Figure 1 is a flow chart of a method for classifying objects in video data according to an embodiment of the present invention. The method may include the following steps:

[0031] Step S101, obtaining video data collected in a target scene.

[0032] In the technical solution provided in step S101 of the present invention, the video data may include at least one video frame data. The target scene may include the following scenes: school monitoring scene, park monitoring scene, and special institution monitoring scene, etc., which are only given as examples and are not specifically limited.

[0033] In this embodiment, video data collected in the target scene is obtained. Optionally, in this embodiment, in the target scene, the image data of the monitoring area is collected in real time using an image acquisition device to obtain video data. Among them, the image acquisition device may include at least one of the following devices or a combination thereof: a camera and a drone, etc. The monitoring area may be an area to be monitored that is adapted to the target scene. For example, if the target scene is a school monitoring scene, the monitoring area may be a teaching area adapted to the school monitoring scene. If the target scene is a park monitoring scene, the monitoring area may be an operating area adapted to the park monitoring scene. If the target scene is a special institution monitoring scene, the monitoring area may be a management area adapted to the special institution monitoring scene. This is only an example and is not specifically limited.

[0034] Step S102 , detecting at least one initial object of the video frame data in the video data to obtain at least one first detection result.

[0035] In the technical solution provided in the above step S102 of the present invention, the above first detection result can be used to at least indicate the initial category to which the initial object belongs, and the association relationship between the objects in the initial object.

[0036] In this embodiment, after acquiring the video data collected in the target scene, at least one initial object of the video frame data in the video data is detected to obtain at least one first detection result. Optionally, based on the acquired video data, this embodiment locates the area where at least one initial object of the video frame data in the video data is located. In the at least one area located above, at least one initial object is detected to obtain at least one first detection result, that is, the initial category to which the at least one initial object belongs can be obtained.

[0037] Optionally, the initial object is detected in the above-mentioned located area to obtain a first detection result. That is, a corresponding candidate frame can be generated in the above-mentioned located area. The initial object is detected using the generated candidate frame to obtain an initial category to which the initial object belongs.

[0038] Step S103: determining a target category to which at least one target object in the video data belongs based on the first detection result.

[0039] In the technical solution provided in the above step S103 of the present invention, the above target object may include an initial object.

[0040] In this embodiment, after detecting at least one initial object of the video frame data in the video data and obtaining at least one first detection result, the target category to which at least one target object of the video data belongs is determined based on the first detection result. Optionally, based on obtaining at least one first detection result, this embodiment can determine the association relationship between the objects in the initial object from the first detection result, wherein the association relationship can be a positional relationship. According to the determined positional relationship, the first detection result is screened to obtain the target category to which at least one target object of the video data belongs.

[0041] Optionally, the first detection result is screened according to the determined positional relationship, and the target category to which at least one target object of the video data belongs can be obtained. That is, if the positional relationship indicates that the first initial object in the initial objects is attached to the second initial object, the first detection result corresponding to the second initial object is screened out from the first detection result, that is, the initial category to which the second initial object belongs in the video frame data is screened out. On the basis of determining the initial category to which the second initial object belongs in the previous video frame data of the video frame data, the above two initial categories are fused to obtain the target category.

[0042] In the above steps S101 to S103 of the present application, when classifying objects in video data, video data collected in the target scene can be obtained. At least one initial object of the video frame data in the acquired video data is detected to obtain at least one first detection result, and based on the obtained first detection result, the target category to which at least one target object of the video data belongs can be determined, thereby achieving the purpose of real-time tracking of objects in change, thereby solving the technical problem of low classification accuracy of objects in video data, and further achieving the technical effect of improving the classification accuracy of objects in video data.

[0043] The above method of this embodiment is further introduced below.

[0044] As an optional implementation method, step S102 detects at least one initial object in the video frame data in the video data to obtain at least one first detection result, including: generating at least one candidate frame in the video frame data; and detecting at least one initial object in multiple candidate frames to obtain at least one first detection result.

[0045] In this embodiment, the candidate box can be used to select an initial object.

[0046] In this embodiment, after acquiring the video data collected in the target scene, at least one candidate frame is generated in the video frame data. Optionally, based on the acquired video data, this embodiment locates the area where at least one initial object of the video frame data in the video data is located. In the at least one area located above, at least one candidate frame can be generated, thereby selecting the initial object.

[0047] In this embodiment, when multiple candidate frames are generated in the video frame data, at least one initial object is detected in the multiple candidate frames to obtain at least one first detection result. Optionally, this embodiment performs category detection on at least one initial object in the multiple candidate frames generated on the basis of generating at least one candidate frame, and can obtain at least one first detection result, that is, the initial category to which the at least one initial object belongs can be obtained.

[0048] As an optional implementation method, at least one initial object is detected in multiple candidate frames to obtain at least one first detection result, including: in response to the multiple candidate frames including candidate frames corresponding to different categories, the multiple candidate frames are merged; in the merged candidate frames, at least one initial object is detected to obtain at least one first detection result.

[0049] In this embodiment, the overlap between the candidate frames corresponding to the above different categories may exceed the target overlap. The overlap between the candidate frames corresponding to the above different categories may be represented by Intersection Over Union (IOU), and the target overlap may be fine-tuned according to the change of the target scene, and the target overlap may also be referred to as a target threshold.

[0050] In this embodiment, after generating at least one candidate frame in the video frame data, in response to the multiple candidate frames including candidate frames corresponding to different categories, the multiple candidate frames are merged. Optionally, this embodiment identifies the generated multiple candidate frames based on the generation of at least one candidate frame, and if it is identified that the generated multiple candidate frames include candidate frames corresponding to different categories, the multiple candidate frames are merged.

[0051] In this embodiment, in response to the multiple candidate frames including candidate frames corresponding to different categories, after merging the multiple candidate frames, at least one initial object is detected in the merged candidate frame to obtain at least one first detection result. Optionally, this embodiment performs category detection on at least one initial object in the merged candidate frame on the basis of merging the candidate frames, and at least one first detection result can be obtained, that is, the initial category to which the at least one initial object belongs can be obtained.

[0052] As an optional implementation method, step S103, based on the first detection result, determines the target category to which at least one target object of the video data belongs, including: determining the association relationship between the first initial object and the second initial object from the first detection result; in response to the association relationship indicating that the first initial object is attached to the second initial object, determining the current candidate features of the second initial object in the video frame data; based on the current candidate features, determining the target category.

[0053] In this embodiment, the at least one initial object may include a first initial object and a second initial object, wherein a first position of the first initial object in the target scene may be different from a second position of the second initial object in the target scene.

[0054] In this embodiment, after detecting at least one initial object of the video frame data in the video data and obtaining at least one first detection result, the association relationship between the first initial object and the second initial object is determined from the first detection result. Optionally, on the basis of obtaining at least one first detection result, this embodiment performs data separation on the obtained first detection result to obtain the association relationship between the first initial object and the second initial object, thereby determining whether the first initial object is attached to the second initial object.

[0055] In this embodiment, the current candidate feature may be a current frame feature. The first initial object may be an accessory object, and the second initial object may be a main object, which is only used as an example and is not specifically limited.

[0056] In this embodiment, after determining the association relationship between the first initial object and the second initial object from the first detection result, in response to the association relationship indicating that the first initial object is attached to the second initial object, the current candidate features that the second initial object has in the video frame data are determined. Optionally, this embodiment, based on determining the association relationship between the first initial object and the second initial object, if the association relationship between the first initial object and the second initial object indicates that the first initial object is attached to the second initial object, then feature extraction is performed on the video frame data to obtain features of the video frame data. From the features of the video frame data, the current candidate features that the second initial object has in the video frame data can be screened out.

[0057] In this embodiment, in response to the association relationship indicating that the first initial object is attached to the second initial object, after determining the current candidate feature that the second initial object has in the video frame data, the target category is determined based on the current candidate feature. Optionally, this embodiment can determine the previous candidate feature that the second initial object has in the previous video frame data of the video frame data based on the determination of the current candidate feature. Based on the determined current candidate feature and the previous candidate feature, the target category to which at least one target object of the video data belongs can be determined.

[0058] As an optional implementation method, determining the target category based on the current candidate feature includes: determining the previous candidate feature that the second initial object has in the previous video frame data of the video frame data; matching the current candidate feature with the previous candidate feature to obtain a matching result; determining matching information between the matching result and the motion state of the second initial object; and determining the target category based on the matching information.

[0059] In this embodiment, the previous candidate feature may be a previous frame feature.

[0060] In this embodiment, in response to the association relationship indicating that the first initial object is attached to the second initial object, after determining the current candidate feature that the second initial object has in the video frame data, the previous candidate feature that the second initial object has in the previous video frame data of the video frame data is determined. Optionally, this embodiment can determine the previous video frame data of the second initial object in the video frame data on the basis of determining the current candidate feature, and perform feature extraction on the previous video frame data to obtain the features of the previous video frame data. From the features of the previous video frame data, the previous candidate feature that the second initial object has in the previous video frame data can be screened out.

[0061] In this embodiment, the above matching result can be used to represent the feature distance between the current candidate feature and the previous candidate feature.

[0062] In this embodiment, after determining the last candidate feature of the second initial object in the last video frame data of the video frame data, the current candidate feature is matched with the last candidate feature to obtain a matching result. Optionally, this embodiment performs distance matching on the determined current candidate feature and the last candidate feature based on the determination of the last candidate feature to obtain a matching result.

[0063] In this embodiment, the motion state can be used to represent the motion state of the subject object. The matching information can be used to represent the matching cost between the feature distance and the motion state.

[0064] In this embodiment, after matching the current candidate feature with the previous candidate feature and obtaining a matching result, the matching information between the matching result and the motion state of the second initial object is determined; based on the matching information, the target category is determined. Optionally, based on the matching result, the embodiment can calculate the matching information between the motion state of the matching result and the second initial object using the Hungarian algorithm, wherein the above algorithm is only used as an example and is not specifically limited. According to the calculated matching information, the first detection result is screened to obtain the target category to which at least one target object of the video data belongs.

[0065] As an optional implementation method, determining the target category based on the matching information includes: in response to the matching information satisfying the target matching information and the tracking of the second initial object has ended, detecting the second initial object in the previous video frame data to obtain a second detection result; determining the target category based on the first detection result and the second detection result.

[0066] In this embodiment, the second detection result may be used to indicate the initial category to which the second initial object belongs.

[0067] In this embodiment, after determining the matching information between the matching result and the motion state of the second initial object, in response to the matching information satisfying the target matching information and the tracking of the second initial object being completed, the second initial object is detected in the previous video frame data to obtain a second detection result. Optionally, on the basis of determining the matching information, the embodiment judges the relationship between the determined matching information and the target matching information to obtain a first judgment result, and judges whether to end the tracking of the second initial object to obtain a second judgment result. If the first judgment result obtained indicates that the determined matching information satisfies the target matching information, and the second judgment result obtained indicates that the tracking of the second initial object is completed, the area where the second initial object is located is located in the previous video frame data. In the area where the second initial object is located, the second initial object is detected to obtain a second detection result.

[0068] In this embodiment, in response to the matching information satisfying the target matching information and the tracking of the second initial object has been completed, the second initial object is detected in the previous video frame data, and after obtaining the second detection result, the target category is determined based on the first detection result and the second detection result. On the basis of obtaining the second detection result, this embodiment performs label fusion on the above-mentioned first detection result and the above-mentioned second detection result to obtain the target category. Optionally, from the first detection result, the first detection result that does not match the second detection result is deleted, and the first detection result that matches the second detection result is retained, and the initial category represented by the first detection result after the above-mentioned processing is determined as the target category.

[0069] In an embodiment of the present invention, when classifying objects in video data, video data collected in a target scene can be obtained. At least one initial object of the video frame data in the acquired video data is detected to obtain at least one first detection result, and according to the obtained first detection result, the target category to which at least one target object of the video data belongs can be determined, thereby achieving the purpose of real-time tracking of objects in change, thereby solving the technical problem of low classification accuracy of objects in video data, and further achieving the technical effect of improving the classification accuracy of objects in video data.

[0070] The technical solution of the embodiment of the present invention is illustrated below in conjunction with preferred implementation modes.

[0071] In the process of processing video data, it often takes a lot of time to track the changing objects. Even if the change track of the object is obtained, it may not be possible to classify the object into the ideal category, which leads to the technical problem of low classification accuracy of the objects in the video data.

[0072] In order to solve the above technical problems, an embodiment of the present invention proposes a method for classifying objects in video data. When classifying objects in video data, video data collected in a target scene can be obtained. At least one initial object of video frame data in the acquired video data is detected to obtain at least one first detection result, and based on the obtained first detection result, the target category to which at least one target object of the video data belongs can be determined, thereby achieving the purpose of real-time tracking of objects in change, thereby solving the technical problem of low classification accuracy of objects in video data, and further achieving the technical effect of improving the classification accuracy of objects in video data.

[0073] In this embodiment, the video parsing method is executed to detect at least one initial object of the video frame data in the acquired video data, and at least one first detection result can be obtained, and according to the obtained first detection result, the target category to which at least one target object of the video data belongs can be determined. For example, FIG2(a) is a flow chart of a video parsing method according to an embodiment of the present invention. As shown in FIG2(a), the method may include the following steps:

[0074] Step S201, input the video frame data into the object detection model to perform object detection, and obtain a three-level label.

[0075] In the technical solution provided in step S201 of the present invention, if the video frame data is input into the parsing device, the object detection model is automatically called. If the user does not set the required label, the parsing device will use the default object category label to perform object detection (also known as target detection), and three-level labels and corresponding candidate boxes can be obtained.

[0076] After obtaining the third-level labels, proceed to step S202 and step S203, perform non-maximum suppression (NMS) processing on the third-level labels to obtain the second-level labels, and fuse the current candidate features to obtain the target image features.

[0077] In the technical solution provided by the above step S202 of the present invention, the candidate frames generated in step S201 are subjected to NMS post-processing, and the candidate frames with IOU greater than the threshold are merged and filtered. In the specific operation of NMS, the target frames belonging to the same coarse-grained category are merged according to the label system. For example, if two labels, man and youth, appear in a picture, and both labels belong to the coarse category label of the human body, the candidate frames corresponding to the two labels will enter the NMS operation process together, and the label corresponding to the retained result of NMS will be used as the detection label of the target frame.

[0078] In the technical solution provided in the above step S203 of the present invention, feature fusion is performed on the current candidate features of multiple initial objects in multiple candidate boxes in the current video frame, that is, the multiple current candidate features corresponding to the initial object before the NMS operation are weightedly fused according to the feature confidence, and the fused result is used as the target image feature corresponding to the initial object.

[0079] After obtaining the secondary label, proceed to step S204 and step S205, determine the obtained secondary label as the detection result of the current frame, and perform target binding on the video frame data, so as to obtain the accessory object and the main object.

[0080] In the technical solution provided in the above step S205 of the present invention, the initial objects existing in the candidate frame filtered in step S201 are subjected to target binding to obtain binding relationship information. If the initial object 1 is completely contained in the initial object 2, and the category definition of the initial object 1 is subordinate to the category of the initial object 2, the binding relationship information indicates that the binding relationship between the initial object 1 and the initial object 2 is established.

[0081] In this embodiment, according to the above binding relationship information, the accessory object can be filtered out, and no subsequent tracking processing is performed on the accessory object. According to the above binding relationship information, the main object can also be filtered out, and the matching features output by the classification head of the main object in the object detection model are obtained, and the above matching features are used as the current frame features of the main target.

[0082] After obtaining the target image features, accessory objects and main object, proceed to step S206 and step S207, perform feature matching on the current frame features of the main target in the target image features with the previous frame features of the main target, and track the main target for the successfully matched main target to obtain the trajectory information of the main target.

[0083] In the technical solution provided in the above step S206 of the present invention, the target image feature is matched with the current candidate feature of the main object, that is, the feature distance between the target image feature and the current candidate feature of the main object is calculated. If the feature distance is less than a certain distance threshold, the target image feature is successfully matched with the current candidate feature of the main object. When the target image feature is successfully matched with the current candidate feature of the main object, step S207 is continued.

[0084] In the technical solution provided in the above step S207 of the present invention, when tracking the main target, feature matching is performed between the current frame feature of the main target and the previous frame feature of the main target, that is, the feature distance between the current frame feature and the previous frame feature is calculated. The motion state of the main target can be predicted by using the calculated and updated Kalman filter, and the feature distance between the current frame feature and the previous frame feature and the motion state of the main object can be optimized and allocated by using the Hungarian algorithm, so as to obtain the matching cost, thereby achieving the purpose of effectively associating the category label with the main object. By minimizing the above matching cost, the minimum matching cost can be obtained, and the main target corresponding to the minimum matching cost is regarded as a successful match.

[0085] In this embodiment, for the subject target that fails to match, a new trajectory is started or the lost trajectory is deleted.

[0086] After the trajectory information of the main target is obtained, the process proceeds to step S208 to perform label fusion on the detection labels of the main target of the video frame data corresponding to the trajectory information, filter out misdetected labels, and archive the correct labels.

[0087] In this embodiment, the feature matching effect may be as shown in FIG2(b). For example, FIG2(b) is a schematic diagram of a feature matching effect according to an embodiment of the present invention. As shown in FIG2(b), word vector feature extraction is performed on the third-level labels, and feature matching is performed on the extracted third-level labels to obtain second-level labels, wherein the third-level labels may include but are not limited to: ragdoll cat, tabby cat and rose, etc., and the second-level labels may include but are not limited to: cat and flower, etc. Word vector feature extraction is performed on the obtained second-level labels, and feature matching is performed on the extracted second-level labels to obtain first-level labels, wherein the first-level labels may include but are not limited to: animals and plants, etc.

[0088] In this embodiment, the affiliation may be as shown in Figure 2(c). For example, Figure 2(c) is a schematic diagram of an affiliation according to an embodiment of the present invention. As shown in Figure 2(c), under the primary label, human organs and clothing accessories belong to the same coarse category label of human body.

[0089] In this embodiment, an object tracking architecture is used to track at least one target object of the video data. For example, FIG2(d) is a schematic diagram of an object tracking architecture according to an embodiment of the present invention. As shown in FIG2(d), the video frame data is encoded by using an image encoder 231, and the encoded video frame data is feature enhanced. The feature-enhanced video frame data is input into a classifier 232 for classification, and the category to which the initial object belongs can be obtained. The feature-enhanced video frame data is input into a detector 233 for target detection, and a candidate frame of the initial object can be obtained. The category label corresponding to the candidate frame is compared with the category label of all things, and the loss between the category labels of the two can be obtained, wherein the category label of all things can be obtained from a text encoder 235. The image corresponding to the candidate frame is input into a feature fusion device 234, and the current frame feature of the main object can be obtained. The feature matching of the current frame feature and the previous frame feature can be performed to obtain the feature distance between the current frame feature and the previous frame feature, and the trajectory matching of the main object is performed using the obtained feature distance between the current frame feature and the previous frame feature, and the trajectory of the main object can be obtained.

[0090] In this embodiment, the label fusion effect can be as shown in Figure 2(e). For example, Figure 2(e) is a schematic diagram of a label fusion effect according to an embodiment of the present invention. As shown in Figure 2(e), the candidate boxes of each video frame data in the video data are respectively subjected to NMS post-processing, and the candidate features of at least one initial object of each video frame data in the candidate box are subjected to feature fusion, so that the target image features can be obtained, and the target image features are used to track all objects of the initial object. According to the order of each video frame data in the video data, the labels corresponding to the candidate boxes of each video frame data obtained by processing are used as the first frame detection label, the second frame detection label...the Nth frame detection label. The above-mentioned frame detection labels are fused to obtain the target archive label.

[0091] In this embodiment, when classifying objects in video data, video data collected in a target scene can be obtained. At least one initial object of the video frame data in the acquired video data is detected to obtain at least one first detection result, and according to the obtained first detection result, the target category to which at least one target object of the video data belongs can be determined, thereby achieving the purpose of real-time tracking of objects in change, thereby solving the technical problem of low classification accuracy of objects in video data, and further achieving the technical effect of improving the classification accuracy of objects in video data.

[0092] According to an embodiment of the present invention, a device for classifying objects in video data is also provided. It should be noted that the device for classifying objects in video data can be used to execute a method for classifying objects in video data in an embodiment.

[0093] Figure 3 FIG. 1 is a schematic diagram of a device for classifying objects in video data according to an embodiment of the present invention. Figure 3 As shown, the device 300 for classifying objects in video data may include: an acquisition unit 301 , a detection unit 302 , and a determination unit 303 .

[0094] The acquisition unit 301 is used to acquire video data collected in a target scene, wherein the video data includes at least one video frame data.

[0095] The detection unit 302 is used to detect at least one initial object of the video frame data in the video data to obtain at least one first detection result, wherein the first detection result is used to at least indicate an initial category to which the initial object belongs.

[0096] The determination unit 303 is configured to determine, based on the first detection result, a target category to which at least one target object in the video data belongs, wherein the target object includes an initial object.

[0097] Optionally, the detection unit 302 may include: a generation module, used to generate at least one candidate frame in the video frame data, wherein the candidate frame is used to select the initial object; a detection module, used to detect at least one initial object in multiple candidate frames to obtain at least one first detection result.

[0098] Optionally, the detection module may include: a merging submodule, used to merge multiple candidate frames in response to multiple candidate frames including candidate frames corresponding to different categories, wherein the overlap between the candidate frames corresponding to different categories exceeds the target overlap; and a detection submodule, used to detect at least one initial object in the merged candidate frames to obtain at least one first detection result.

[0099] Optionally, the determination unit 303 may include: a first determination module, used to determine the association relationship between the first initial object and the second initial object from the first detection result, wherein at least one initial object includes the first initial object and the second initial object, and a first position of the first initial object in the target scene is different from a second position of the second initial object in the target scene; a second determination module, used to determine the current candidate features of the second initial object in the video frame data in response to the association relationship indicating that the first initial object is attached to the second initial object; and a third determination module, used to determine the target category based on the current candidate features.

[0100] Optionally, the third determination module may include: a first determination submodule, used to determine the previous candidate feature of the second initial object in the previous video frame data of the video frame data; a matching submodule, used to match the current candidate feature with the previous candidate feature to obtain a matching result; a second determination submodule, used to determine the matching information between the matching result and the motion state of the second initial object, wherein the matching information is used to represent the matching cost between the matching result and the motion state; and a third determination submodule, used to determine the target category based on the matching information.

[0101] Optionally, the third determination submodule can complete the determination of the target category based on the matching information by executing the following steps: in response to the matching information satisfying the target matching information and the tracking of the second initial object has been completed, the second initial object is detected in the previous video frame data to obtain a second detection result, wherein the second detection result is used to indicate the initial category to which the second initial object belongs; based on the first detection result and the second detection result, the target category is determined.

[0102] In this embodiment, an acquisition unit is used to acquire video data collected in a target scene, wherein the video data includes at least one video frame data; a detection unit is used to detect at least one initial object of the video frame data in the video data to obtain at least one first detection result, wherein the first detection result is used to at least represent an initial category to which the initial object belongs; a determination unit is used to determine, based on the first detection result, a target category to which at least one target object of the video data belongs, wherein the target object includes the initial object, thereby achieving the purpose of real-time tracking of changing objects, thereby solving the technical problem of low classification accuracy of objects in video data, and further achieving the technical effect of improving the classification accuracy of objects in video data.

[0103] According to an embodiment of the present invention, a processor is further provided. The processor is used to run a program. When the program is run by the processor, the method for classifying objects in video data in the embodiment is executed.

[0104] According to an embodiment of the present invention, an electronic device is further provided, comprising: a memory storing an executable program; and a processor for running the program, wherein the method for classifying objects in video data in the embodiment is executed when the program is running.

[0105] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method for classifying objects in video data in the embodiment.

[0106] According to an embodiment of the present invention, a computer program product is further provided. The computer program product includes a computer program. When the computer program is executed by a processor, the method for classifying objects in video data in the embodiment is implemented.

[0107] According to an embodiment of the present invention, a computer program product is also provided, including a non-volatile computer-readable storage medium, the non-volatile computer-readable storage medium is used to store a computer program, and when the computer program is executed by a processor, the method for classifying objects in video data in the embodiment is implemented.

[0108] According to an embodiment of the present invention, a computer program is further provided. When the computer program is executed by a processor, the method for classifying objects in video data in the embodiment is implemented.

[0109] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0110] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0111] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0112] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed over multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0113] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0114] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the relevant technology or the whole or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, referred to as Read-Only Memory), random access memory (RAM, referred to as Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0115] The above are only preferred embodiments of the present invention. It should be pointed out that, for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for classifying objects in video data, characterized in that: include: Acquire video data collected in a target scene, wherein the video data includes at least one video frame data; Detecting at least one initial object of the video frame data in the video data to obtain at least one first detection result, wherein the first detection result is used to at least indicate an initial category to which the initial object belongs; Based on the first detection result, a target category to which at least one target object of the video data belongs is determined, wherein the target object includes the initial object.

2. The method according to claim 1, characterized in that Detecting at least one initial object of the video frame data in the video data to obtain at least one first detection result includes: In the video frame data, generating at least one candidate frame, wherein the candidate frame is used to select the initial object; In the plurality of candidate frames, at least one of the initial objects is detected to obtain at least one first detection result.

3. The method according to claim 2, characterized in that In the plurality of candidate frames, detecting at least one of the initial objects to obtain at least one first detection result includes: In response to the plurality of candidate frames including candidate frames corresponding to different categories, merging the plurality of candidate frames, wherein the overlap between the candidate frames corresponding to the different categories exceeds a target overlap; In the merged candidate frame, at least one of the initial objects is detected to obtain at least one first detection result.

4. The method according to claim 1, characterized in that Determining, based on the first detection result, a target category to which at least one target object of the video data belongs, includes: Determine, from the first detection result, an association relationship between a first initial object and a second initial object, wherein at least one of the initial objects includes the first initial object and the second initial object, and a first position of the first initial object in the target scene is different from a second position of the second initial object in the target scene; In response to the association relationship indicating that the first initial object is attached to the second initial object, determining a current candidate feature of the second initial object in the video frame data; Based on the current candidate features, the target category is determined.

5. The method according to claim 4, characterized in that Determining the target category based on the current candidate features includes: Determine a previous candidate feature that the second initial object has in the previous video frame data of the video frame data; Matching the current candidate feature with the previous candidate feature to obtain a matching result; Determining matching information between the matching result and the motion state of the second initial object, wherein the matching information is used to represent a matching cost between the matching result and the motion state; Based on the matching information, the target category is determined.

6. The method according to claim 5, characterized in that Determining the target category based on the matching information includes: In response to the matching information satisfying the target matching information and the tracking of the second initial object has ended, detecting the second initial object in the previous video frame data to obtain a second detection result, wherein the second detection result is used to indicate the initial category to which the second initial object belongs; The target category is determined based on the first detection result and the second detection result.

7. A device for classifying objects in video data, characterized in that: include: An acquisition unit, configured to acquire video data collected in a target scene, wherein the video data includes at least one video frame data; a detection unit, configured to detect at least one initial object of the video frame data in the video data to obtain at least one first detection result, wherein the first detection result is used to at least indicate an initial category to which the initial object belongs; A determination unit is used to determine, based on the first detection result, a target category to which at least one target object of the video data belongs, wherein the target object includes the initial object.

8. A processor, characterized in that: The processor is used to run a program, wherein the program, when run by the processor, executes the method for classifying objects in video data as claimed in any one of claims 1 to 6.

9. An electronic device, characterized in that: include: A memory storing an executable program; A processor is used to run the program, wherein the program, when running, executes the method for classifying objects in video data as described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored executable program, wherein when the executable program is run, the device where the storage medium is located is controlled to execute the method for classifying objects in video data as described in any one of claims 1 to 6.

Images