A processing method and a processing device
By acquiring video data and motion trajectory signals of the target object, processing them to generate second video data, and labeling the object to be labeled based on the transformation relationship, the problem of inaccurate vehicle positioning in camera footage is solved, and the accuracy of labeling is improved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- LENOVO (BEIJING) LTD
- Filing Date
- 2022-09-30
- Publication Date
- 2026-07-24
AI Technical Summary
In existing technologies, vehicle positioning in camera footage is inaccurate, especially for vehicles without OBU devices, resulting in inaccurate location information and significant errors in labeling information.
By acquiring video data and motion trajectory signals of the target object, second video data is generated using structured data information processing. Based on the motion trajectory signals and structured data information, the object to be labeled is labeled, and the conversion relationship between pixel position and GPS position is established to improve the labeling accuracy.
It enables accurate positioning of objects to be labeled in the camera view, improving the accuracy of location information and labeling precision.
Smart Images
Figure CN115601716B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of intelligent transportation technology, and in particular to a processing method and processing device. Background Technology
[0002] Currently, the position information of vehicles in the camera view is predicted by mapping the 3D point cloud coordinates of specific objects in the LiDAR to the 2D coordinates in the camera view. However, there are certain limitations to using LiDAR, such as the possibility of losing track of the target and the insufficient accuracy of predicting GPS information.
[0003] While some vehicles can provide GPS location information, in real-world applications, it's impossible for every vehicle appearing in the camera's view to be equipped with an OBU device.
[0004] Therefore, in the existing technology, when locating and labeling vehicles in camera footage, there is a problem that the location information of some vehicles cannot be accurately located, resulting in large errors in the labeling information. Summary of the Invention
[0005] This application provides a processing method and a processing apparatus. The technical solutions adopted in the embodiments of this application are as follows:
[0006] On the one hand, embodiments of this application provide a processing method, including:
[0007] If the target object meets the preset conditions, the first video data and motion trajectory signal of the target object are acquired;
[0008] Structured data information of the moving object is obtained from the first video data, and the structured data information is used to process the first video data to generate second video data; wherein, the moving object includes a target object and an object to be labeled;
[0009] The objects to be labeled in the second video data are labeled based on the motion trajectory signal and the structured data information.
[0010] In some embodiments, obtaining structured data information of the moving object based on the first video data includes:
[0011] Frame by frame, the first video data is identified to obtain the identification information of all moving objects and their corresponding bounding boxes;
[0012] Each of the aforementioned identification information and its corresponding bounding box is saved accordingly to obtain the structured data information of each of the moving objects.
[0013] In some embodiments, labeling the objects to be labeled in the second video data based on the motion trajectory signal and the structured data information includes:
[0014] The target bounding box set is determined from the structured data information based on the motion trajectory signal and the target object identification information;
[0015] The transformation relationship between position information and bounding boxes is determined based on the motion trajectory signal and the target bounding box set;
[0016] The second location information of the object to be labeled is determined based on the transformation relationship, so as to perform the labeling.
[0017] In some embodiments, determining a set of target bounding boxes from the structured data information based on the motion trajectory signal and the target object identification information includes:
[0018] A first set of timestamps is obtained based on the motion trajectory signal;
[0019] The second timestamp set corresponding to each first bounding box is determined from the structured data information using the timestamp information corresponding to the video frames and the target object identification information;
[0020] Based on the first timestamp set, the second timestamp set is registered, and the bounding boxes corresponding to the timestamps with the smallest deviation from the first timestamp are taken as the target bounding box set.
[0021] In some embodiments, determining the transformation relationship between position information and bounding boxes based on the motion trajectory signal and the target bounding box set includes:
[0022] A first set of position information is obtained based on the motion trajectory signal;
[0023] Based on the first set of location information and the set of target bounding boxes, a dataset of point pairs corresponding to the location information and the bounding boxes is obtained;
[0024] Based on the point pair dataset, a homography matrix is obtained to transform the location information and the bounding box.
[0025] In some embodiments, determining the second location information of the object to be labeled based on the transformation relationship includes:
[0026] Determine the identification information of the object to be labeled, and determine the second bounding box of the video frame from the structured data information based on the identification information of the object to be labeled;
[0027] The second location information of the object to be labeled is determined based on the second bounding box and the homography matrix, and the identification information of the object to be labeled is associated with the second location information.
[0028] In some embodiments, the preset conditions include:
[0029] The location information of the target object indicates that the target object has entered the preset area; or, based on the detection of the object to be labeled by the target object.
[0030] On the other hand, embodiments of this application also provide a processing apparatus, including:
[0031] The acquisition module is configured to acquire the first video data and motion trajectory signal of the target object when the target object meets preset conditions.
[0032] The video data processing module is configured to: obtain structured data information of the moving object based on the first video data, and process the first video data using the structured data information to generate second video data; wherein the moving object includes a target object and an object to be labeled;
[0033] The annotation module is configured to annotate the objects to be annotated in the second video data based on the motion trajectory signal and the structured data information.
[0034] In some embodiments, the video data processing module includes:
[0035] The identification unit is configured to identify the moving object based on video frames of the first video data, and obtain the identification information of the moving object and the corresponding bounding boxes.
[0036] The storage unit is configured to save the identification information and each of the bounding boxes correspondingly to obtain the structured data information.
[0037] In some embodiments, the annotation module includes:
[0038] The determining unit determines a set of target bounding boxes from the structured data information based on the motion trajectory signal and the target object identification information;
[0039] The calculation unit is configured to: determine the transformation relationship between position information and bounding boxes based on the motion trajectory signal and the target bounding box set.
[0040] The annotation unit is configured to determine the second location information of the object to be annotated based on the transformation relationship, so as to perform the annotation.
[0041] Compared with the prior art, the beneficial effects of the embodiments of this application are as follows: When the target object meets preset conditions, the embodiments of this application acquire video data and motion trajectory signals of the target object; by processing the video data, structured data information of all moving objects, including the target object and the object to be labeled, is obtained; and the object to be labeled in the second video data is labeled based on the motion trajectory signals and the structured data information. In this embodiment, by using the location information of the real scene provided by the target object and the pixel location information provided by the video data, the conversion relationship between the pixel location information in the video data and the location information in the real scene can be obtained. Therefore, the location information of the object to be labeled in the real scene can be predicted based on the pixel location information of the object to be labeled in the video data, improving the accuracy of labeling the object to be labeled in the video data. Attached Figure Description
[0042] Figure 1 A flowchart of the processing method provided in the embodiments of this application;
[0043] Figure 2 A flowchart illustrating the processing method provided in an embodiment of this application;
[0044] Figure 3 Another schematic flowchart of the processing method provided in the embodiments of this application;
[0045] Figure 4 This is yet another flowchart illustrating the processing method provided in the embodiments of this application;
[0046] Figure 5 A block diagram of the processing apparatus provided in the embodiments of this application;
[0047] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0048] Various embodiments and features of this application are described herein with reference to the accompanying drawings.
[0049] It should be understood that various modifications can be made to the embodiments described herein. Therefore, the above description should not be considered as limiting, but merely as an example of embodiments. Other modifications within the scope and spirit of this application will be apparent to those skilled in the art.
[0050] The accompanying drawings, which are included in and form part of this specification, illustrate embodiments of the present application and, together with the general description of the present application given above and the detailed description of the embodiments given below, serve to explain the principles of the present application.
[0051] These and other features of this application will become apparent from the following description of preferred forms of embodiments given as non-limiting examples, with reference to the accompanying drawings.
[0052] It should also be understood that although this application has been described with reference to some specific examples, those skilled in the art can certainly implement many other equivalent forms of this application, which have the features described in the claims and are therefore all within the scope of protection defined herein.
[0053] The above and other aspects, features and advantages of this application will become more apparent when taken in conjunction with the accompanying drawings and in view of the following detailed description.
[0054] Specific embodiments of this application are described thereafter with reference to the accompanying drawings; however, it should be understood that the claimed embodiments are merely examples of this application, which can be implemented in various ways. Well-known and / or repeated functions and structures are not described in detail to avoid unnecessary or redundant details that could obscure the application. Therefore, the specific structural and functional details claimed herein are not intended to be limiting, but merely serve as the basis and representative basis for the claims to teach those skilled in the art to use this application in a variety of substantially any suitable detailed structures.
[0055] This specification may use the phrases “in one embodiment,” “in another embodiment,” “in yet another embodiment,” or “in other embodiments,” all of which may refer to one or more of the same or different embodiments according to this application.
[0056] Figure 1 A flowchart illustrating the processing method provided in an embodiment of this application is shown. Figure 1 As shown, the processing method of this application embodiment includes the following steps S100-S300:
[0057] S100: If the target object meets the preset conditions, acquire the first video data and motion trajectory signal of the target object.
[0058] This step aims to acquire the motion trajectory information and corresponding video data of the target object. In this embodiment, the target object can be a mobile object that can provide its trajectory information in real time, such as a target vehicle equipped with an On-Board Unit (OBU), or a mobile object that can provide real-time GPS information through other devices. Preset conditions can be set according to the video data acquisition time or the detection status of the object to be labeled. For example, when the target object enters a preset area, such as when a target vehicle equipped with an OBU enters the camera's shooting range, the target vehicle can be considered to meet the preset conditions. Then, the motion trajectory signal is acquired through the target vehicle's OBU; simultaneously, the shooting device acquires first video data based on the target vehicle to obtain the first video data of the target object and its associated motion trajectory signal. The preset conditions can also be that the subsequent acquisition steps are performed when the object to be labeled is selected as the object to be detected. The selection method can be a common method in the art, such as interface selection, special tags, etc., which will not be described in detail here.
[0059] In some specific applications, this embodiment can determine whether a target vehicle meets preset conditions by acquiring GPS location information reported by the On-Board Unit (OBU). When the GPS location information indicates that the target vehicle has entered the shooting range of a camera, it is considered that the target vehicle meets the preset conditions, triggering the camera to start recording. The camera begins recording video of the target vehicle and ends recording when the target vehicle leaves the shooting range, thus obtaining the first video data of the target vehicle. Of course, the camera can also maintain recording mode, acquiring video data for all time periods, and then extracting a portion of the video data for the corresponding time period as the first video data based on whether the target object meets the preset conditions, such as the time periods when it enters and leaves the shooting range. This application does not limit this. Based on the corresponding situation of the target vehicle meeting the preset conditions, such as the time periods when it enters and leaves the shooting range, the GPS location information reported by the On-Board Unit (OBU) is used to obtain the motion trajectory signal corresponding to the vehicle passing through the shooting range.
[0060] For example, Figure 2 The flowchart illustrating the data acquisition process of the processing method provided in the embodiments of this application in some practical applications is shown below. See also... Figure 2As shown, in some practical applications, during data acquisition, the GPS location information reported by the On-Board Unit (OBU) determines whether the target vehicle has entered the camera's shooting area. If it has not entered the shooting area, the GPS location information of the target vehicle continues to be collected. If the GPS location information reported by the OBU indicates that the target vehicle has entered the camera's shooting area, the camera is triggered to start video streaming and recording. During recording, the GPS location information reported by the OBU determines whether the target vehicle has left the camera's shooting area. If the GPS location information reported by the OBU indicates that the target vehicle has left the shooting area, recording ends to obtain the first video data of the target vehicle. During video recording, the GPS location information collected and reported by the OBU and the corresponding timestamps are recorded in the database. Recording stops when the GPS location information reported by the OBU indicates that the target vehicle has left the shooting area to obtain the target vehicle's motion trajectory signal.
[0061] S200: Obtain structured data information of the moving object based on the first video data, and process the first video data using the structured data information to generate second video data; wherein, the moving object includes a target object and an object to be labeled.
[0062] This step aims to process the acquired first video data to identify moving objects and obtain corresponding structured data information. Based on this structured data information and the first video data, second video data is generated to facilitate subsequent annotation of objects to be labeled based on the target objects. In this step, moving objects can include the target object and other objects to be labeled, such as a target vehicle equipped with an On-Board Unit (OBU) and other vehicles without an OBU. This allows for the annotation of other vehicles using GPS location information provided by the target vehicle equipped with an OBU. The structured data information can include the identification number of the moving object and its corresponding bounding box, such as a rectangle. For example, AI recognition processing can be performed frame-by-frame on the first video data to detect all vehicles in the video frame. Each vehicle is named using preset rules to determine its identification ID. Then, the rectangle information of all vehicles in each video frame is detected, allowing the location information of all vehicles in the video frame to be marked using rectangles. Finally, the vehicle identification IDs and the rectangle information from each video frame are extracted, and the identification IDs of the same vehicle are associated with and saved with each rectangle to obtain the structured data information of all vehicles. Then, the structured data information is used to process each video frame of the first video data, draws a corresponding rectangle for each vehicle in the video frame, adds an identification ID to the rectangle, and writes it into the first video data frame by frame. After completion, the second video data is generated.
[0063] In some embodiments, obtaining structured data information of the moving object based on the first video data can be implemented as follows: steps S210-S220:
[0064] S210, Identify the first video data to obtain the identification information of all moving objects and their corresponding bounding boxes;
[0065] S220, each of the identification information and the corresponding bounding box are saved to obtain the structured data information of each of the moving objects.
[0066] This embodiment aims to perform frame-by-frame recognition processing on each video frame of the first video data to obtain structured data information of all moving objects within it. In some specific implementations of this embodiment, the recognition and detection of the first video data can be performed frame-by-frame using a preset vehicle detection model to identify vehicles in the first video data and detect all vehicle information. During the recognition process, different detected vehicles can be identified using preset rules. When the same vehicle is detected, it is named using the identification ID (Track ID) identified during the first detection. Finally, the identification ID (Track ID) information of each vehicle in all video frames is obtained. Simultaneously, the bounding box (BBOX) information corresponding to each vehicle in the video frame is determined based on the vehicle detection model, so that the position information of the corresponding vehicle in the video frame can be identified through the bounding box. Of course, based on the type of object to be labeled, other moving object detection models can also be used to detect moving objects in the first video data to determine all moving objects and their corresponding bounding boxes. Finally, the identification ID information and bounding boxes of moving objects in each video frame are extracted, and the identification ID information of each moving object and the bounding boxes of the moving object in each video frame are saved in a corresponding association, thereby obtaining the structured data information of all moving objects.
[0067] For example, Figure 3 This is another flowchart illustrating the process of processing video data using the processing method provided in this application in some practical applications. See also... Figure 3As shown, in some practical applications, when identifying and detecting the first video data, AI recognition and analysis methods can be used to process the first video data frame by frame. Pre-set model information is used to identify moving objects, and each moving object is named according to pre-set rules to determine its Track ID. The bounding box (BBOX) information of each moving object in each video frame is also determined. Finally, the Track ID of the moving object and its corresponding BBOX information in all video frames are saved to obtain the structured data information of each moving object. This structured data information is then used to process the first video data frame by frame, drawing corresponding BBOXes for each moving object in the video frame and adding Track IDs to the rectangles. After adding these BBOXes frame by frame, the AI-processed second video data is obtained.
[0068] S300, the objects to be labeled in the second video data are labeled based on the motion trajectory signal and the structured data information.
[0069] This step aims to utilize the correspondence between the motion trajectory signal and structured data information of the target object to determine its GPS location information using the structured data information of the object to be labeled, thereby labeling the object in the second video data. In this step, to determine the GPS location information of the object to be labeled, a conversion relationship between GPS location information and pixel location information in the image can be calculated based on the motion trajectory signal and structured data information of the target object. Then, the GPS location information of the object to be labeled can be calculated based on this conversion relationship using the structured data information of the object to be labeled.
[0070] In some embodiments, labeling the objects to be labeled in the second video data based on the motion trajectory signal and the structured data information includes:
[0071] The target bounding box set is determined from the structured data information based on the motion trajectory signal and the target object identification information;
[0072] The transformation relationship between position information and bounding boxes is determined based on the motion trajectory signal and the target bounding box set;
[0073] The second location information of the object to be labeled is determined based on the transformation relationship, so as to perform the labeling.
[0074] In this embodiment, to determine the conversion relationship between GPS location information and pixel location information in an image, the corresponding bounding boxes are determined from structured data information using the target object's identification information. Then, based on the target object's motion trajectory signal, target bounding boxes corresponding one-to-one with the GPS location information are determined from each bounding box, resulting in the target bounding box set. Next, based on the location information in the target object's motion trajectory signal and the pixel box location information corresponding to the target bounding box set, the conversion relationship between the GPS location information and the corresponding bounding boxes is determined. Therefore, based on this conversion relationship, the corresponding GPS location information of the object to be labeled can be determined using its bounding box information for appropriate labeling.
[0075] In some embodiments, a set of target bounding boxes is determined from the structured data information based on the motion trajectory signal and the target object identification information, so as to determine the target bounding boxes that correspond one-to-one with the GPS location information. Specific steps include:
[0076] A first set of timestamps is obtained based on the motion trajectory signal;
[0077] The second timestamp set corresponding to each first bounding box is determined from the structured data information using the timestamp information corresponding to the video frames and the target object identification information;
[0078] Based on the first timestamp set, the second timestamp set is registered, and the bounding boxes corresponding to the timestamps with the smallest deviation from the first timestamp are taken as the target bounding box set.
[0079] In this embodiment, considering the time stamp error between the GPS location information reported by the OBU mounted on the target vehicle and the timestamps of each video frame in the second video data, a first timestamp set corresponding to the motion trajectory signal is obtained by analyzing the GPS log file reported by the OBU mounted on the target vehicle to obtain the timestamp information of each GPS location information. Then, the timestamp information of each video frame is obtained based on the second video data, and the timestamp information of each video frame corresponding to each bounding box of the target object is determined based on the target object identification information to obtain the second timestamp set. Afterwards, using the time registration principle, the second timestamp set is registered based on the first timestamp set. Each second timestamp in the second timestamp set is compared with each first timestamp in the first timestamp set. The bounding box corresponding to the second timestamp with the smallest timestamp error is registered with the GPS location information under the corresponding first timestamp to obtain the target bounding box corresponding to the GPS location information. Thus, the bounding boxes corresponding to the timestamps with the smallest deviation from the first timestamp are taken as the target bounding box set. Here, the threshold for timestamp error can be determined based on the actual situation or the experience of technical personnel, so as to delete GPS location information and corresponding target bounding boxes when the timestamp error exceeds the threshold, and ensure the accuracy of subsequent calculations of transformation relationships.
[0080] In some embodiments, determining the transformation relationship between position information and bounding boxes based on the motion trajectory signal and the target bounding box set includes:
[0081] A first set of position information is obtained based on the motion trajectory signal;
[0082] Based on the first set of location information and the set of target bounding boxes, a dataset of point pairs corresponding to the location information and the bounding boxes is obtained;
[0083] Based on the point pair dataset, a homography matrix is obtained to transform the location information and the bounding box.
[0084] This embodiment aims to calculate the conversion relationship between bounding boxes and GPS location information based on motion trajectory signals and the registered bounding boxes. In this embodiment, GPS location information can be obtained from the GPS log files reported by the OBU (On-Board Unit) of the target vehicle to obtain a first location information set. Then, based on the first location information set and the corresponding registered target bounding box set, a point-to-point dataset (gps, pixels) between the location information and the bounding boxes is obtained. Based on this point-to-point dataset, a homography matrix between the pixel positions in the video frame and the GPS location information in the real scene is calculated. This homography matrix can then be used to annotate other objects in the video data.
[0085] In some embodiments, determining the second location information of the object to be labeled based on the transformation relationship includes:
[0086] Determine the identification information of the object to be labeled, and determine the second bounding box of the video frame from the structured data information based on the identification information of the object to be labeled;
[0087] The second location information of the object to be labeled is determined based on the second bounding box and the homography matrix, and the identification information of the object to be labeled is associated with the second location information.
[0088] This embodiment aims to determine the corresponding bounding box information based on the annotation information of the object to be annotated, and to determine the GPS location information of the object to be annotated based on the bounding box information and the homography matrix calculated previously. In this embodiment, the object to be annotated can be specified by a technician, or it can be another moving object in the second video data besides the target object; this application does not limit this. In some specific applications, when annotating one of the other moving objects in the second video data besides the target object, the identification information of the object to be annotated can be determined first. For example, the identification information of the object to be annotated can be the identification ID information identified in the aforementioned AI recognition process, so that the corresponding bounding box information can be determined based on the structured data information through the identification ID information. Calculations based on the second bounding box information and the homography matrix can obtain the second location information of the object to be annotated, that is, the GPS location information in the real scene corresponding to the object to be annotated when it is in the video frame corresponding to the second bounding box information. Thus, the calculated GPS location information can be associated with the identification information of the object to be annotated. Of course, according to the settings of the technician, the GPS location information and the identification information of the object to be annotated can also be presented in the video frame corresponding to the second bounding box for easy and intuitive display.
[0089] For example, Figure 4 The flowchart illustrating the processing method provided in the embodiments of this application in some practical applications is shown below. See also Figure 4As shown, in some practical applications, when labeling objects in the AI-processed second video data, the homography matrix between the pixel position information and the GPS position information in the real scene is first determined based on the pixel position information of the target vehicle equipped with an OBU device in each video frame and the GPS position information in the corresponding trajectory signal. Then, the homography matrix is combined with the pixel position information of the object to be detected in the video frame to calculate and determine the GPS position corresponding to the object in the video frame, so as to label the object to be detected in each video frame. In some specific applications, in order to determine the pixel position information of the target vehicle in the video frame, the Track ID of the target vehicle is first extracted, and then the bounding box (BBOX) information corresponding to the target vehicle is obtained from the structured data information based on the Track ID. After that, the GPS log file reported by the OBU installed on the target vehicle is obtained to obtain the motion trajectory signal corresponding to the second video data. Based on the motion trajectory signal, the first timestamp set corresponding to each GPS position of the target vehicle can be obtained. Furthermore, based on the second video data, the timestamp information corresponding to each video frame is obtained. The timestamp information of each bounding box (BBOX) of the target vehicle is then used to add corresponding timestamp information, resulting in a second set of timestamps corresponding to each bounding box of the target vehicle. Next, using the time registration principle (i.e., time matching principle), the second set of timestamps is registered using the first set of timestamps. This involves comparing the timestamp information corresponding to each BBOX with the timestamp information corresponding to each GPS location. From all BBOXes corresponding to the target vehicle, the BBOXes with the smallest timestamp error relative to each GPS location are determined. After registering and aligning the corresponding BBOX information with the GPS location information, a BBOX / GPS point pair dataset is obtained. Based on this point pair dataset (BBOX / GPS), the homography matrix between the pixel positions in the video frame and the GPS location information in the real scene is calculated. Therefore, for other objects to be labeled in the second video data, the homography matrix calculated based on the target vehicle can be used to determine the corresponding GPS location information through the bounding box information (BBOX) of other objects to be labeled, allowing for appropriate labeling of these objects in the second video data. In this embodiment, when calculating the homography matrix, the smaller the timestamp error between the BBOX information and the GPS location information during the alignment process, the smaller the GPS distance error of the object to be labeled can be reduced, thereby improving the accuracy of GPS calculation and prediction.
[0090] In some embodiments, the preset conditions include: the location information of the target object indicates that the target object has entered a preset area; or, based on the fact that the target object detects the object to be labeled.
[0091] This embodiment aims to determine the preset conditions that a target object must meet, so as to obtain the corresponding first video data and motion trajectory signal based on the preset conditions. In this embodiment, the preset condition is set as the location information of the target object indicating that the target object enters a preset area. The preset area can be the shooting range of the camera capable of video capture. In some specific applications, the location information of the target vehicle can be obtained by reporting GPS location information through the on-board unit (OBU). When the GPS location information indicates that the target vehicle has entered the shooting range of a camera, it is considered that the target vehicle meets the preset condition. At this time, the first video data of the target vehicle when it is within the shooting range can be obtained through the camera, and the GPS location information of the target vehicle during the time period when it is within the shooting range can be reported through the on-board unit (OBU) to obtain the motion trajectory signal of the target vehicle.
[0092] In this embodiment, the preset condition is that the object to be labeled is detected based on the target object. The object to be labeled can be other vehicles or other moving objects that need to be labeled, detected based on image recognition. In some specific applications, image recognition and detection can be performed on the captured image including the target vehicle. When other vehicles that need to be labeled are detected, the target vehicle is considered to meet the preset condition. At this time, video is recorded by the camera until the other vehicles to be labeled leave the camera's shooting range, thereby obtaining the first video data of the target vehicle. At the same time, the GPS location information reported by the on-board unit (OBU) is used to obtain the motion trajectory signal of the target vehicle corresponding to the shooting time, which is used for subsequent labeling of the object to be labeled.
[0093] Based on the same inventive concept, embodiments of this application provide a processing apparatus. Figure 5 A block diagram of a processing apparatus provided in an embodiment of this application. Figure 5 As shown, the processing device includes:
[0094] The acquisition module 10 is configured to acquire the first video data and motion trajectory signal of the target object when the target object meets preset conditions.
[0095] The video data processing module 20 is configured to: obtain structured data information of a moving object based on the first video data, and process the first video data using the structured data information to generate second video data; wherein the moving object includes a target object and an object to be labeled;
[0096] The annotation module 30 is configured to annotate the objects to be annotated in the second video data based on the motion trajectory signal and the structured data information.
[0097] In some embodiments, the video data processing module includes:
[0098] The identification unit is configured to identify the moving object based on video frames of the first video data, and obtain the identification information of the moving object and the corresponding bounding boxes.
[0099] The storage unit is configured to save the identification information and each of the bounding boxes correspondingly to obtain the structured data information.
[0100] In some embodiments, the annotation module includes:
[0101] The determining unit determines a set of target bounding boxes from the structured data information based on the motion trajectory signal and the target object identification information;
[0102] The calculation unit is configured to: determine the transformation relationship between position information and bounding boxes based on the motion trajectory signal and the target bounding box set.
[0103] The annotation unit is configured to determine the second location information of the object to be annotated based on the transformation relationship, so as to perform the annotation.
[0104] In conjunction with any of the embodiments described herein, the acquisition module 10, video data processing module 20, annotation module 30, and their respective necessary functional units included in the processing apparatus of each embodiment of this application can be further configured to implement the steps of the corresponding processing methods described above.
[0105] This application also provides an electronic device, including at least a memory 901, a processor 902, and a bus (not shown), wherein the structural schematic diagram of the electronic device can be as follows: Figure 6 As shown, the memory 901 stores machine-readable instructions that can be executed by the processor 902. When the electronic device is running, the processor 902 communicates with the memory 901 via a bus. When the machine-readable instructions are executed by the processor, they perform the steps of the processing method provided in any embodiment of this application.
[0106] Since the electronic device described in this application embodiment is an electronic device equipped with a memory for implementing the processing method disclosed in this application embodiment, those skilled in the art can understand the structure and variations of the electronic device described in this application embodiment based on the processing method described in this application embodiment, and therefore will not be described again here.
[0107] This application also provides a storage medium that stores one or more programs. When the one or more programs are executed by a processor, they implement the steps of the processing method provided in any embodiment of this application.
[0108] The storage medium in this embodiment may be included in an electronic device / system; or it may exist independently and not assembled into an electronic device / system. The storage medium stores one or more programs, which, when executed, implement the steps of the processing method provided in the embodiments of this application.
[0109] According to embodiments of this application, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. Optionally, specific examples in this embodiment can refer to the examples described in any embodiment of this application, which will not be repeated here. Obviously, those skilled in the art should understand that the various modules or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, this application is not limited to any particular hardware and software combination.
[0110] The above embodiments are merely exemplary embodiments of this application and are not intended to limit this application. The scope of protection of this application is defined by the claims. Those skilled in the art can make various modifications or equivalent substitutions to this application within its substance and scope of protection, and such modifications or equivalent substitutions should also be considered to fall within the scope of protection of this application.
Claims
1. A processing method, wherein, include: If the target object meets the preset conditions, the first video data and motion trajectory signal of the target object are acquired; Based on the first video data, structured data information of the moving object is obtained, and the structured data information is used to process the first video data to generate second video data; wherein, the moving object includes a target object and an object to be labeled; the structured data information includes the identification information of the moving object and its corresponding bounding box; The annotation of the objects to be annotated in the second video data based on the motion trajectory signal and the structured data information specifically includes: determining a set of target bounding boxes from the structured data information based on the motion trajectory signal and the target object identification information; determining the transformation relationship between position information and bounding boxes based on the motion trajectory signal and the set of target bounding boxes; and determining the second position information of the objects to be annotated based on the transformation relationship for the annotation. The process of determining a set of target bounding boxes from the structured data information based on the motion trajectory signal and the target object identification information includes: A first set of timestamps is obtained based on the motion trajectory signal; The second timestamp set corresponding to each first bounding box is determined from the structured data information using the timestamp information corresponding to the video frames and the target object identification information; Based on the first timestamp set, the second timestamp set is registered, and the bounding boxes corresponding to the timestamps with the smallest deviation from the first timestamp are taken as the target bounding box set.
2. The method according to claim 1, wherein, The structured data information of the moving object obtained from the first video data includes: Frame by frame, the first video data is identified to obtain the identification information of all moving objects and their corresponding bounding boxes; Each of the aforementioned identification information and its corresponding bounding box is saved accordingly to obtain the structured data information of each of the moving objects.
3. The method according to claim 1, wherein, Determining the transformation relationship between position information and bounding boxes based on the motion trajectory signal and the target bounding box set includes: A first set of position information is obtained based on the motion trajectory signal; Based on the first set of location information and the set of target bounding boxes, a dataset of point pairs corresponding to the location information and the bounding boxes is obtained; Based on the point pair dataset, a homography matrix is obtained to transform the location information and the bounding box.
4. The method according to claim 3, wherein, Determining the second location information of the object to be labeled based on the transformation relationship includes: Determine the identification information of the object to be labeled, and determine the second bounding box of the video frame from the structured data information based on the identification information of the object to be labeled; The second location information of the object to be labeled is determined based on the second bounding box and the homography matrix, and the identification information of the object to be labeled is associated with the second location information.
5. The method according to any one of claims 1-4, wherein, The preset conditions include: The location information of the target object indicates that the target object has entered the preset area; or, based on the detection of the object to be labeled by the target object.
6. A processing apparatus, wherein, include: The module is configured as follows: If the target object meets the preset conditions, the first video data and motion trajectory signal of the target object are acquired; The video data processing module is configured to: obtain structured data information of a moving object based on the first video data, and process the first video data using the structured data information to generate second video data; wherein, the moving object includes a target object and an object to be labeled; the structured data information includes the identification information of the moving object and its corresponding bounding box; The annotation module is configured to annotate the objects to be annotated in the second video data based on the motion trajectory signal and the structured data information; The annotation module includes: The determining unit determines a set of target bounding boxes from the structured data information based on the motion trajectory signal and the target object identification information; The calculation unit is configured to: determine the transformation relationship between position information and bounding boxes based on the motion trajectory signal and the target bounding box set. The annotation unit is configured to determine the second location information of the object to be annotated based on the transformation relationship, so as to perform the annotation; The process of determining a set of target bounding boxes from the structured data information based on the motion trajectory signal and the target object identification information includes: A first set of timestamps is obtained based on the motion trajectory signal; The second timestamp set corresponding to each first bounding box is determined from the structured data information using the timestamp information corresponding to the video frames and the target object identification information; Based on the first timestamp set, the second timestamp set is registered, and the bounding boxes corresponding to the timestamps with the smallest deviation from the first timestamp are taken as the target bounding box set.
7. The processing apparatus according to claim 6, wherein, The video data processing module includes: The identification unit is configured to identify the moving object based on video frames of the first video data, and obtain the identification information of the moving object and the corresponding bounding boxes. The storage unit is configured to save the identification information and each of the bounding boxes correspondingly to obtain the structured data information.