Method, device, equipment and storage medium for determining association relationship

Abstract

The embodiments of the present disclosure disclose a method, apparatus, device and storage medium for determining an association relationship, wherein the method includes: obtaining multiple trajectory segments corresponding to a video to be processed and a spliced ​​image corresponding to each trajectory segment; the spliced ​​image includes image information of multiple image frames of a first person object corresponding to the trajectory segment in the video to be processed; obtaining a first annotation result for the spliced ​​image corresponding to each trajectory segment; the first annotation result is used to characterize whether the first person object and the second person object corresponding to the trajectory segment are the same; for each trajectory segment, based on the first annotation result of the spliced ​​image corresponding to the trajectory segment, adding a second annotation result to the multiple trajectory image frames corresponding to the trajectory segment, the second annotation result is used to determine whether there is a person object that has an association relationship with the target object in the multiple trajectory image frames corresponding to the trajectory segment.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing, and more particularly to a method, apparatus, device, and storage medium for determining an association relationship. Background Art

[0002] To ensure safety in various scenarios, real-time video streams must be monitored. If a person carrying an illegal object appears in the scene, a corresponding alert message must be generated. Related technologies require manual recognition of large numbers of image frames to determine the association between the illegal object and the person carrying the illegal object. This is not only inefficient but also carries a certain error rate. Summary of the Invention

[0003] Embodiments of the present disclosure provide a method, apparatus, device, and storage medium for determining an association relationship.

[0004] In a first aspect, a method for determining an association relationship is provided, comprising:

[0005] Acquire multiple trajectory segments corresponding to the video to be processed and a spliced ​​image corresponding to each of the trajectory segments; the spliced ​​image includes image information of multiple image frames of the first person object corresponding to the trajectory segment in the video to be processed;

[0006] Obtaining a first annotation result for the stitched image corresponding to each of the trajectory segments; the first annotation result is used to indicate whether a first person object and a second person object corresponding to the trajectory segment are the same; the second person object is a person object associated with the target object in the video to be processed;

[0007] For each of the trajectory segments, based on the first annotation result of the spliced ​​image corresponding to the trajectory segment, a second annotation result is added to the multiple trajectory image frames corresponding to the trajectory segment, and the second annotation result is at least used to determine whether there is a human object associated with the target object in the multiple trajectory image frames corresponding to the trajectory segment.

[0008] In some embodiments, obtaining a plurality of trajectory segments corresponding to the video to be processed and a stitched image corresponding to each of the trajectory segments includes:

[0009] performing a first detection process and a tracking process on the video to be processed for a person object, to obtain a trajectory segment corresponding to each of a plurality of first person objects; the trajectory segment including a plurality of trajectory image frames and position information of the first person object in each of the trajectory image frames;

[0010] For each of the trajectory segments, a spliced ​​image corresponding to the trajectory segment is generated based on a plurality of trajectory image frames corresponding to the trajectory segment and position information of the first human object in each of the trajectory image frames.

[0011] In the embodiment of the present disclosure, since the video to be processed is divided into multiple trajectory segments based on detection processing and tracking processing, clustering of all trajectory image frames corresponding to the video to be processed with human objects is achieved, that is, the same trajectory segment includes trajectory image frames of the same human object, which provides a theoretical basis for subsequently determining the annotation information of all trajectory image frames by using the annotation information of the spliced ​​image.

[0012] In some embodiments, generating a spliced ​​image corresponding to the trajectory segment based on the plurality of trajectory image frames corresponding to the trajectory segment and the position information of the first person object in each of the trajectory image frames includes:

[0013] Screening the multiple trajectory image frames to obtain multiple image frames to be spliced;

[0014] generating a first person image corresponding to each of the image frames to be stitched based on the plurality of image frames to be stitched and position information of the first person object in each of the image frames to be stitched;

[0015] The spliced ​​image is generated based on the first person image corresponding to each of the image frames to be spliced.

[0016] In the embodiment of the present disclosure, since multiple trajectory image frames are first screened and a corresponding stitched image is generated based on the screened multiple image frames to be stitched, the probability that the stitched image includes a human object and a target object having an association relationship can be increased, thereby improving the overall labeling efficiency and accuracy.

[0017] In some embodiments, performing first detection and tracking processing on the video to be processed for a person object to obtain at least one trajectory segment corresponding to the first person object includes:

[0018] Performing a first detection process on the video to be processed for human objects to obtain a first detection result for each scene image frame in the video to be processed; the first detection result includes position information corresponding to each first human object in the scene image frame;

[0019] Performing a first detection process on the plurality of scene image frames for human objects to obtain a first detection result for each of the scene image frames, wherein the first detection result includes position information corresponding to each of the first human objects in the scene image frames;

[0020] Based on the frame number of each of the scene image frames and the position information corresponding to each of the first human objects, a trajectory segment corresponding to each of the first human objects is generated.

[0021] In the embodiment of the present disclosure, since the spliced ​​image includes the character features of the first character object corresponding to the trajectory segment and the first character images corresponding to multiple image frames to be spliced ​​in the video to be processed, the accuracy of the first annotation result can be improved.

[0022] In some embodiments, obtaining a first annotation result for the stitched image corresponding to each of the trajectory segments includes:

[0023] For each of the trajectory segments, based on the spliced ​​image corresponding to the trajectory segment, obtaining a first person feature of a first person object corresponding to the trajectory segment;

[0024] Acquire a second person feature of the second person object in a reference image; the reference image is an image frame in the video to be processed that represents an association relationship between the second person object and the target object;

[0025] The first labeling result is generated based on the first character feature and the second character feature.

[0026] In some embodiments, obtaining a first annotation result for the stitched image corresponding to each of the trajectory segments includes:

[0027] For each of the trajectory segments, a stitched image and a reference image corresponding to the trajectory segment are displayed through a first interface; the reference image is an image frame in the video to be processed that represents an association relationship between the second person object and the target object;

[0028] A first annotation operation on the stitched image is received to obtain a first annotation result.

[0029] In the embodiment of the present disclosure, since the reference image is an image frame in the video to be processed that represents the association relationship between the second person object and the target object, the first annotation result of the spliced ​​image determined based on the reference image is more accurate.

[0030] In some embodiments, the method for obtaining the reference image includes at least one of the following:

[0031] displaying a plurality of candidate reference images through a second interface; receiving a selection operation for a target candidate reference image among the plurality of candidate reference images, and determining the target candidate reference image as the reference image;

[0032] Obtain a frame number corresponding to each of the multiple candidate reference images; determine a target frame number based on the frame number corresponding to each of the candidate reference images, and determine the candidate reference image corresponding to the target frame number as the reference image.

[0033] In some embodiments, the video to be processed includes multiple scene image frames, and the multiple reference images to be selected are obtained in at least one of the following ways:

[0034] Determining the multiple scene image frames as the multiple reference images to be selected;

[0035] performing a second detection process on the plurality of scene image frames for the target object, and using the scene image frames including the target object as the plurality of reference images to be selected;

[0036] performing a second detection process on the plurality of scene image frames for the target object to obtain at least one scene image frame including the target object and a second detection result corresponding to each of the scene image frames, and determining the plurality of candidate reference images based on the second detection result corresponding to each of the scene image frames;

[0037] Perform a second detection process on the multiple scene image frames for the target object to obtain a second detection result corresponding to each of the scene image frames, wherein the second detection result includes position information of the target object in the scene image frame; perform a first detection process on the multiple scene image frames for the human object to obtain a first detection result for each of the scene image frames, wherein the first detection result includes position information corresponding to each of the first human objects in the scene image frame; determine the multiple reference images to be selected from the multiple scene image frames based on the first detection result and the second detection result corresponding to each of the scene image frames.

[0038] In the embodiment of the present disclosure, since the reference image is obtained by screening multiple candidate reference images, it is more likely that the person object in the reference image is a second person object that has an associated relationship with the target object, thereby improving the accuracy of the first annotation result.

[0039] In some embodiments, adding a second annotation result to the plurality of trajectory image frames corresponding to the trajectory segment based on the first annotation result of the spliced ​​image corresponding to the trajectory segment includes:

[0040] When the first labeling result indicates that the first person object corresponding to the trajectory segment and the second person object are the same, obtaining a plurality of trajectory image frames corresponding to the trajectory segment;

[0041] Add the second annotation results to the multiple trajectory image frames; the second annotation results include at least one of the following: the first annotation result, position information of the first person object corresponding to the trajectory segment in each of the trajectory image frames, the frame number of each of the trajectory image frames, and the trajectory identifier corresponding to the trajectory segment.

[0042] In some embodiments, obtaining a first annotation result for the stitched image corresponding to each of the trajectory segments includes:

[0043] For each first annotation result of the spliced ​​image corresponding to the trajectory segment, a verification result of the first annotation result is obtained, and the first annotation result is adjusted based on the verification result.

[0044] In the embodiment of the present disclosure, since the first annotation result is added to the multiple trajectory image frames corresponding to the trajectory segment, the position information of the first person object corresponding to the trajectory segment in each of the trajectory image frames, the frame number of each of the trajectory image frames and the trajectory identifier corresponding to the trajectory segment can be used to determine the association relationship between the illegal object and the person object carrying the illegal object, and at the same time, the position information and trajectory information of the person object of the illegal object in the trajectory image frame can be determined. Compared with the solution of manual identification in the related art, it can not only save a lot of workload, but also obtain richer annotation information.

[0045] In a second aspect, a device for determining an association relationship is provided, comprising:

[0046] A first acquisition module is configured to acquire a plurality of trajectory segments corresponding to a video to be processed and a spliced ​​image corresponding to each of the trajectory segments; the spliced ​​image includes image information of a plurality of image frames of a first person object corresponding to the trajectory segment in the video to be processed;

[0047] a second acquisition module configured to acquire a first annotation result for the spliced ​​image corresponding to each of the trajectory segments; the first annotation result being used to indicate whether the first and second person objects corresponding to the trajectory segments are identical; the second person object being a person object associated with the target object in the video to be processed;

[0048] The annotation module is configured to, for each of the trajectory segments, add a second annotation result to the multiple trajectory image frames corresponding to the trajectory segment based on the first annotation result of the stitched image corresponding to the trajectory segment, wherein the second annotation result is at least used to determine whether there is a human object associated with the target object in the multiple trajectory image frames corresponding to the trajectory segment.

[0049] In a third aspect, an association relationship determination device is provided, comprising: a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and the processor implements the steps in the above method when executing the computer program.

[0050] In a fourth aspect, a computer storage medium is provided, wherein the computer storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the above method.

[0051] In the embodiment of the present disclosure, since a stitched image corresponding to each trajectory segment in the video to be processed is obtained, and a second stitched annotation result is added to all trajectory video frames in the trajectory segment using the first annotation result corresponding to the stitched image, compared with the solution in the related art that manually identifies a large number of image frames to determine the association relationship between illegal objects and human objects carrying illegal objects, not only the annotation efficiency but also the accuracy can be improved; at the same time, since the stitched image includes the human features of the first human object corresponding to the trajectory segment in multiple image frames in the video to be processed, the accuracy of the first annotation result can be improved; at the same time, since the first annotation result corresponding to the stitched image is used as the second annotation result for all trajectory video frames in the trajectory segment, compared with the solution in the related art that manually determines the association relationship between illegal objects and human objects carrying illegal objects in each image frame, not only the annotation task in complex scenes can be achieved, but also the problem of manual erroneous labeling caused by one-sided observation can be reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 A flowchart of a method for determining an association relationship provided in an embodiment of the present disclosure;

[0053] Figure 2 A flowchart of a method for determining an association relationship provided in an embodiment of the present disclosure;

[0054] Figure 3 A flowchart of a method for determining an association relationship provided in an embodiment of the present disclosure;

[0055] Figure 4 A flowchart of a method for determining an association relationship provided in an embodiment of the present disclosure;

[0056] Figure 5 A schematic diagram of a video preprocessing method according to an embodiment of the present disclosure;

[0057] Figure 6 A schematic diagram of two trajectory segments and corresponding stitched images provided in an embodiment of the present disclosure;

[0058] Figure 7A flowchart of a method for preparing labeled data provided in an embodiment of the present disclosure;

[0059] Figure 8 A flowchart of a data annotation method provided in an embodiment of the present disclosure;

[0060] Figure 9 A flowchart of a method for processing annotation results provided by an embodiment of the present disclosure;

[0061] Figure 10 A schematic diagram of the structure of an apparatus for determining an association relationship provided by an embodiment of the present disclosure;

[0062] Figure 11 A schematic diagram of a hardware entity of an association relationship determination device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0063] The following embodiments and accompanying drawings will be used to describe the technical solutions of the present disclosure in detail. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0064] It should be noted that in the examples of this disclosure, "first", "second", etc. are used to distinguish similar objects, and are not necessarily used to describe the order or precedence of the objects. In addition, the technical solutions described in the embodiments of this disclosure can be arbitrarily combined without conflict.

[0065] The embodiment of the present disclosure provides a method for determining association relationships, which can improve the efficiency and accuracy of annotation, thereby improving the efficiency and accuracy of determining association relationships as a whole. The method for determining association relationships provided by the embodiment of the present disclosure is applied to electronic devices.

[0066] The following describes exemplary applications of the electronic devices provided by the embodiments of the present disclosure. The electronic devices provided by the embodiments of the present disclosure can be implemented as various types of user terminals (hereinafter referred to as terminals), such as laptop computers, tablet computers, desktop computers, set-top boxes, mobile devices (for example, mobile phones, portable music players, personal digital assistants, dedicated messaging devices, portable gaming devices), and can also be implemented as servers.

[0067] See also Figure 1 , Figure 1 This is an optional flow chart of the method for determining the association relationship provided by the embodiment of the present disclosure, which will be combined with Figure 1 The steps shown are explained.

[0068] S101. Acquire multiple trajectory segments corresponding to a video to be processed and a spliced ​​image corresponding to each trajectory segment; the spliced ​​image includes image information of multiple image frames of a first person object corresponding to the trajectory segment in the video to be processed.

[0069] In some embodiments, the video to be processed may be derived from video data collected in real time by a camera device, or may be video data pre-stored in a server. That is, after the camera device collects the video data, the server may process the video data in real time, or it may be pre-stored and then processed. For example, the processing may be performed when the server has fewer processing tasks, or the processing may be performed at a time specified by the operator. Thus, in the acquisition of video data, it may be video data currently collected by the camera device, or it may be video data pre-stored in a server, that is, video data collected by a camera device at a historical time may be retrieved, which is not limited here.

[0070] In some embodiments, the camera equipment can be deployed in various places, such as a corner of the ceiling inside a building, a wall outside a building, or inside an elevator device. Accordingly, the video to be processed can be any image in any scene where the camera equipment is deployed, any image inside a building, or any image outside a building, and there is no limitation here.

[0071] In some embodiments, the trajectory segments corresponding to the video to be processed are achieved using a target detection algorithm and a tracking algorithm for human objects. The target detection algorithm and the tracking algorithm may be pre-set by the server before acquiring the video to be processed. After acquiring the video to be processed, the server may process the video to be processed based on the pre-set target detection algorithm and the tracking algorithm to obtain multiple trajectory segments corresponding to the video to be processed.

[0072] It should be noted that, for each trajectory segment, the trajectory segment includes multiple trajectory image frames arranged in time sequence. Taking the video to be processed as an example, the multiple trajectory image frames corresponding to the trajectory segment can also be arranged according to the frame number of each trajectory image frame. The frame number of the trajectory image frame is the order of the scene image frame corresponding to the trajectory image frame in the video to be processed.

[0073] In some embodiments, for any trajectory segment corresponding to a first human object, the trajectory segment may include all video frames in which the first human object appears in the video to be processed. Through all video frames in which the first human object appears, all human features of the first human object in the current camera device (video to be processed) can be determined. In this embodiment, in order to reduce the amount of data calculation and improve data processing efficiency, a spliced ​​image corresponding to the trajectory segment can be obtained, and the human features of the first human object in the spliced ​​image can be equivalent to all human features of the first human object in the current camera device (video to be processed); wherein the spliced ​​image is obtained by splicing multiple image frames of the first human object in the video to be processed; and the multiple image frames can be all or part of the image frames in the video to be processed.

[0074] S102. Obtain a first annotation result for the stitched image corresponding to each of the trajectory segments; the first annotation result is used to indicate whether a first person object and a second person object corresponding to the trajectory segment are the same; the second person object is a person object in the video to be processed that has an associated relationship with the target object.

[0075] In some embodiments, the target object can be a real object such as an animal, a building, a plant or a daily necessity in a real scene. In an implementation scenario of the embodiment of the present disclosure, the target object is an illegal object corresponding to the real scene, and the real scene is the real scene corresponding to the video to be processed (the real scene captured by the corresponding camera device). When the real scene is located inside a building, for example, when the real scene is inside an elevator, the target object can be a non-motor vehicle; when the real scene is a student dormitory, the target object can be a high-power device such as a hair dryer or an electric rice cooker; when the real scene is located outside a building, for example, when the real scene is a gas station, the target object can be various fire sources such as lighters; when the real scene is an airport, the target object can be a drone.

[0076] In some embodiments, the target object exists in at least one video frame of the video to be processed, and the second person object is a person object in the at least one video frame that is associated with the target object. The second person object can be obtained by at least one of the following methods:

[0077] Obtained by manual annotation;

[0078] obtaining position information of the target object in the image frame by using an image detection method for the target object, and obtaining position information of at least one human object by using an image detection method for the human object, and determining a second human object associated with the target object based on the position information of the target object in the image frame and the position information of the at least one human object;

[0079] The position information of the target object in the image frame is obtained by an image detection method for the target object; based on the position information of the target object in the image frame, a target human body part associated with the target object is determined, and the target human body part is the human body part closest to the target object in the image frame; based on the target human body part, a second human object in the image frame that has an associated relationship with the target object is determined.

[0080] In some embodiments, the first annotation result may be obtained by manual annotation; the first annotation result may also be determined by comparing the character features of the first character object with the character features of the second character object.

[0081] S103. For each trajectory segment, based on the first annotation result of the stitched image corresponding to the trajectory segment, add a second annotation result to the multiple trajectory image frames corresponding to the trajectory segment, where the second annotation result is at least used to determine whether there is a human object associated with the target object in the multiple trajectory image frames corresponding to the trajectory segment.

[0082] In some embodiments, for each trajectory segment, since the multiple trajectory image frames corresponding to the trajectory segment all include character features corresponding to the same first character object, that is, the multiple trajectory image frames all display the first character object, therefore, when the first annotation result of the stitched image corresponding to the trajectory segment can represent whether the first character object and the second character object are the same, the first annotation result corresponding to the stitched image can represent whether all the trajectory image frames corresponding to the trajectory segment include the character features of the first character object that are the same as the second character object, that is, the first annotation result corresponding to the stitched image can represent whether all the trajectory image frames corresponding to the trajectory segment display the first character object that is the same as the second character object.

[0083] It should be noted that, for any trajectory segment corresponding to a first character object, the multiple trajectory image frames corresponding to the trajectory segment include the character features of the first character object in various postures and actions. If the multiple image frames are manually labeled, there will inevitably be a posture and action that makes it difficult for humans to distinguish the first character object that has an association with the target object under the current posture and action; or, the first character object that has an association with the target object is farther away from the target object, and the first character object that has no association with the target object is closer to the target object, which leads to the problem of incorrect marking of the association relationship between the target object and the character object.

[0084] In some implementation scenarios, after obtaining multiple trajectory image frames carrying the second annotation results, the multiple trajectory image frames can be used as a sample set, and a network model can be trained based on the sample set. The network model can be used to detect human objects in the video stream that have an associated relationship with the target object.

[0085] In other implementation scenarios, the embodiments of the present disclosure can obtain at least one image frame associated with the target object for a video to be processed, and then generate a corresponding alarm message based on the at least one image frame to ensure the safety of the current real scene.

[0086] In the embodiment of the present disclosure, since a stitched image corresponding to each trajectory segment in the video to be processed is obtained, and a second stitched annotation result is added to all trajectory video frames in the trajectory segment using the first annotation result corresponding to the stitched image, compared with the solution in the related art that manually identifies a large number of image frames to determine the association relationship between illegal objects and human objects carrying illegal objects, not only the annotation efficiency but also the accuracy can be improved; at the same time, since the stitched image includes the human features of the first human object corresponding to the trajectory segment in multiple image frames in the video to be processed, the accuracy of the first annotation result can be improved; at the same time, since the first annotation result corresponding to the stitched image is used as the second annotation result for all trajectory video frames in the trajectory segment, compared with the solution in the related art that manually determines the association relationship between illegal objects and human objects carrying illegal objects in each image frame, not only the annotation task in complex scenes can be achieved, but also the problem of manual erroneous labeling caused by one-sided observation can be reduced.

[0087] See also Figure 2 , Figure 2 This is an optional flow chart of the method for determining the association relationship provided by the embodiment of the present disclosure, based on Figure 1 , Figure 1 S101 can be included as S201 to S202, which will be combined Figure 2 The steps shown are explained.

[0088] S201. Perform first detection and tracking processing on the video to be processed for human objects to obtain a trajectory segment corresponding to each of the multiple first human objects; the trajectory segment includes multiple trajectory image frames and position information of the first human object in each of the trajectory image frames.

[0089] For each trajectory image frame, the position information may be the position of the detection frame of the first person object in the trajectory image frame. For example, the position information may be the relative position of the vertex of the detection frame of the first person object in the trajectory image frame, or the relative position of the center point of the detection frame of the first person object in the trajectory image frame. This disclosure is not limited to this.

[0090] In some embodiments, the first detection process is for detecting people in an image and can be implemented using any person detection algorithm or model. The tracking process is used to determine a trajectory segment of the people based on the positions of the people in each scene image frame obtained by the first detection process and the temporal relationship between the scene image frames.

[0091] In some embodiments, the above-mentioned first detection processing and tracking processing for the human object in the video to be processed can be implemented through S2011 to S2012 to obtain a trajectory segment corresponding to each of the multiple first human objects.

[0092] S2011. Perform a first detection process on multiple scene image frames of the video to be processed for human objects to obtain a first detection result for each scene image frame in the video to be processed; the first detection result includes position information corresponding to each first human object in the scene image frame.

[0093] The first detection process for human objects can be implemented by a pre-trained human detection model, which is used to detect human objects in an input image and generate position information of each human object in the input image.

[0094] In an embodiment of the present disclosure, the scene image frame may not contain any human object, and accordingly, the first detection result corresponding to the scene image frame does not include the position information of the first human object; the scene image frame may include at least one human object, and accordingly, the first detection result corresponding to the scene image frame may include the position information of each human object.

[0095] S2012: Determine a trajectory segment corresponding to each first human object based on the frame number of each scene image frame and the position information corresponding to each first human object using a tracking algorithm.

[0096] S202 : For each trajectory segment, generate a spliced ​​image corresponding to the trajectory segment based on a plurality of trajectory image frames corresponding to the trajectory segment and position information of the first human object in each trajectory image frame.

[0097] In some embodiments, S2021 to S2023 may be used to implement the above-mentioned generation of a spliced ​​image corresponding to the trajectory segment based on a plurality of trajectory image frames corresponding to the trajectory segment and the position information of the first person object in each of the trajectory image frames.

[0098] S2021 : Filter the multiple trajectory image frames to obtain multiple image frames to be spliced.

[0099] In some embodiments, the plurality of image frames to be spliced ​​may be randomly selected from the plurality of trajectory image frames. In other implementations, the plurality of image frames to be spliced ​​may be selected based on the frame number corresponding to each of the plurality of trajectory image frames. The plurality of image frames to be spliced ​​may be evenly distributed among the plurality of trajectory image frames.

[0100] S2022: Generate a first person image corresponding to each of the image frames to be stitched based on the plurality of image frames to be stitched and position information of the first person object in each of the image frames to be stitched.

[0101] In some embodiments, the first person image can be captured from the image frame to be stitched based on the position information of the image frame to be stitched. When the position information is detection frame information, the first person image can be directly captured based on the detection frame information; when the position information is a center point, a capture frame can be generated based on the coordinates of the center point, and the first person image can be captured from the image frame to be stitched based on the capture frame.

[0102] S2023: Generate the stitched image based on the first person image corresponding to each of the image frames to be stitched.

[0103] In some embodiments, the plurality of image frames to be stitched may be arranged in sequence into a stitched image according to the order of the frame numbers corresponding to each image frame to be stitched, for example, from left to right, from top to bottom, or the like.

[0104] In the embodiment of the present disclosure, since the spliced ​​image includes the character features of the first character object corresponding to the trajectory segment and the first character images corresponding to multiple image frames to be spliced ​​in the video to be processed, the accuracy of the first annotation result can be improved.

[0105] See also Figure 3 , Figure 3This is an optional flow chart of the method for determining the association relationship provided by the embodiment of the present disclosure, based on any of the above embodiments, based on Figure 1 For example, Figure 1 S102 in the above example can be updated to S301 to S303, or to S304 to S305, which will be combined with Figure 3 The steps shown are explained.

[0106] S301 : For each trajectory segment, based on a spliced ​​image corresponding to the trajectory segment, obtain a first person feature of a first person object corresponding to the trajectory segment.

[0107] In some embodiments, a pre-trained feature extraction algorithm / model may be used to extract the first person feature of the first person object in the image to be stitched.

[0108] Among them, since the image to be spliced ​​includes multiple first person images, the first person features corresponding to each first person image can be extracted separately, and the multiple first person features obtained can be fused, and the fused first task features are determined as the first person features of the first person object in the image to be spliced.

[0109] S302. Obtain a second character feature of the second character object in a reference image; the reference image is an image frame in the video to be processed that represents an association relationship between the second character object and the target object.

[0110] The reference image contains a target object, and the target object is associated with a second person object in the reference image. For example, if the target object is a non-motorized vehicle, the reference image may be an image frame in the video to be processed showing a pedestrian A pushing a non-motorized vehicle, with the pedestrian A being the second person object.

[0111] S303: Generate the first annotation result based on the first character feature and the second character feature.

[0112] In some embodiments, the feature distance between the first character feature and the second task feature can be calculated. When the feature distance is less than a feature distance threshold, a first labeling result is generated, indicating that the first character object and the second character object are the same character object; when the feature distance is greater than or equal to the feature distance threshold, a first labeling result is generated, indicating that the first character object and the second character object are not the same character object.

[0113] S304 . For each of the trajectory segments, display a stitched image and a reference image corresponding to the trajectory segment through a first interface; the reference image is an image frame in the video to be processed that represents an association relationship between the second person object and the target object.

[0114] S305: Receive a first annotation operation on the stitched image to obtain the first annotation result.

[0115] A first annotation operation of the user on the image to be spliced ​​may be received to obtain a corresponding first annotation result.

[0116] In some embodiments, the reference image can be obtained by: displaying multiple candidate reference images through a second interface; receiving a selection operation for a target candidate reference image among the multiple candidate reference images, and determining the target candidate reference image as the reference image.

[0117] In some embodiments, the reference image can also be obtained in the following manner: obtaining the frame number corresponding to each of the multiple reference images to be selected; determining the target frame number based on the frame number corresponding to each of the reference images to be selected, and determining the reference image to be selected corresponding to the target frame number as the reference image.

[0118] Based on the above embodiment, the video to be processed includes multiple scene image frames, and the multiple reference images to be selected are obtained in at least one of the following ways:

[0119] Determining the multiple scene image frames as the multiple reference images to be selected;

[0120] performing a second detection process on the plurality of scene image frames for the target object, and using the scene image frames including the target object as the plurality of reference images to be selected;

[0121] Perform a second detection process on the multiple scene image frames for the target object to obtain at least one scene image frame including the target object and a second detection result corresponding to each of the scene image frames, and determine the multiple reference images to be selected based on the second detection result corresponding to each of the scene image frames.

[0122] The second detection result is used to characterize the confidence that the scene image frame includes the target object, and the scene image frame with a confidence higher than a preset confidence threshold is used as a reference image to be selected.

[0123] Perform a second detection process on the multiple scene image frames for the target object to obtain a second detection result corresponding to each of the scene image frames, wherein the second detection result includes position information of the target object in the scene image frame; perform a first detection process on the multiple scene image frames for the human object to obtain a first detection result for each of the scene image frames, wherein the first detection result includes position information corresponding to each of the first human objects in the scene image frame; determine the multiple reference images to be selected from the multiple scene image frames based on the first detection result and the second detection result corresponding to each of the scene image frames.

[0124] Among them, for each scene image frame, the distance between the target object and each first human object can be determined based on the position information corresponding to each first human object in the scene image frame and the position information of the target object in the scene image frame, and the distance between the smallest target object and the first human object is used as the minimum distance corresponding to the scene image frame. When the minimum distance is less than the preset human distance threshold, the scene image frame is used as a reference image to be selected.

[0125] In some embodiments, after obtaining a first annotation result for the stitched image corresponding to each of the trajectory segments, the method further includes: obtaining a verification result for the first annotation result for the stitched image corresponding to each of the trajectory segments, and adjusting the first annotation result based on the verification result.

[0126] In an embodiment of the present disclosure, after obtaining a first annotation result of the stitched image corresponding to each trajectory segment, a manual verification result of the first annotation result may be obtained, and the first annotation result corresponding to the stitched image may be adjusted based on the verification result.

[0127] For example, if there is a first annotation result corresponding to a trajectory segment A, which is that the first character object (male student wearing a hat) in the trajectory segment A and the second character object are the same character object, that is, the first character object (male student wearing a hat) in the trajectory segment A and the target object are associated with each other, then the spliced ​​image and the reference image corresponding to the trajectory segment A can be displayed through a visual interface, and the user's verification result for the first annotation result can be received. If the verification result indicates that the current first annotation result is correct, the first annotation result corresponding to the spliced ​​image is not adjusted. If the verification result indicates that the current first annotation result is incorrect, the first annotation result corresponding to the spliced ​​image is adjusted to indicate that the first character object (male student wearing a hat) in the trajectory segment A and the second character object are not the same character object.

[0128] In the embodiment of the present disclosure, since the reference image is an image frame in the video to be processed that represents the association between the second human object and the target object, the first annotation result of the spliced ​​image determined based on the reference image is correspondingly more accurate; at the same time, since the reference image is obtained by screening multiple reference images to be selected, the possibility that the human object in the reference image is the second human object that has an association with the target object can be made higher, thereby improving the accuracy of the first annotation result.

[0129] See also Figure 4 , Figure 4 This is an optional flow chart of the method for determining the association relationship provided by the embodiment of the present disclosure, based on any of the above embodiments, based on Figure 1 For example, Figure 1 S104 in can be updated to S401 to S402, which will be combined Figure 4 The steps shown are explained.

[0130] S401 : When the first annotation result indicates that the first person object and the second person object corresponding to the trajectory segment are the same, obtain a plurality of trajectory image frames corresponding to the trajectory segment.

[0131] S402. Add the second annotation results to the multiple trajectory image frames; the second annotation results include at least one of the following: the first annotation result, position information of the first person object corresponding to the trajectory segment in each of the trajectory image frames, the frame number of each of the trajectory image frames, and the trajectory identifier corresponding to the trajectory segment.

[0132] In the embodiment of the present disclosure, for the multiple trajectory image frames corresponding to the trajectory segment, a corresponding second annotation result may be added to each trajectory image frame.

[0133] In some embodiments, the second annotation result includes the first annotation result, and the first annotation result is used to indicate that there is a person object in the trajectory image frame that has an association relationship with the target object.

[0134] In some embodiments, the second annotation result includes a frame number of the trajectory image frame, which is used to represent the temporal relationship of the trajectory image frame in the trajectory segment. Exemplarily, the frame number can be the temporal number of the trajectory image frame among multiple scene image frames in the video to be processed, or the temporal number of the trajectory image frame among multiple trajectory image frames corresponding to the trajectory segment.

[0135] In some embodiments, the second annotation result includes position information of the first person object corresponding to the trajectory segment in each of the trajectory image frames, and the position information is used to determine the position information of the first person object associated with the target object in the trajectory image frame.

[0136] In some embodiments, the second annotation result includes a trajectory identifier corresponding to the trajectory segment. In the process of obtaining at least one trajectory segment corresponding to the video to be processed, the trajectory identifier corresponding to each trajectory segment can also be obtained. In the case of tracking loss, in a video to be processed, the first person object associated with the target object can correspond to multiple interval trajectory segments, and the multiple interval trajectory segments respectively correspond to different trajectory identifiers, and there are no overlapping trajectory image frames between the multiple interval trajectory segments. Generally speaking, there is often a loss interval between the multiple interval trajectory segments when tracking is lost. Therefore, in order to reflect this scenario, it is necessary to add the trajectory identifier corresponding to the current trajectory segment to the trajectory image frame corresponding to each trajectory segment.

[0137] In some implementation scenarios, when the multiple trajectory image frames are used as sample sets and a network model is trained based on the sample sets, sample sets with different requirements can be obtained based on the above scheme, thereby realizing network model training tasks for different scenarios.

[0138] In the embodiment of the present disclosure, since the first annotation result is added to the multiple trajectory image frames corresponding to the trajectory segment, the position information of the first person object corresponding to the trajectory segment in each of the trajectory image frames, the frame number of each of the trajectory image frames and the trajectory identifier corresponding to the trajectory segment can be used to determine the association relationship between the illegal object and the person object carrying the illegal object, and at the same time, the position information and trajectory information of the person object of the illegal object in the trajectory image frame can be determined. Compared with the solution of manual identification in the related art, it can not only save a lot of workload, but also obtain richer annotation information.

[0139] The following describes an exemplary application of the embodiments of the present application in a practical application scenario.

[0140] With the rise in urbanization, community safety is drawing increasing attention. Non-motorized vehicles entering elevators and residential buildings can easily cause explosions or fires, posing a significant safety hazard. Developing a computer vision-based algorithm for detecting and associating non-motorized vehicles in elevators is crucial. When a resident pushes a non-motorized vehicle into an elevator, this algorithm can generate an alarm, associate the non-motorized vehicle with the owner, and immediately report the alarm to the property management, facilitating safety education and accountability for the owner.

[0141] However, building the algorithm requires data annotation of the collected elevator videos to generate a high-quality test dataset to validate the algorithm's association results. Traditional methods directly decode the video, detect faces, bodies, and non-motorized vehicles, and annotate each frame with the human and non-motorized objects. This annotation process is very expensive and presents the following challenges: high annotation costs; long annotation cycles; and the sheer volume of images makes annotation prone to errors. Overall, from an economic perspective, traditional annotation methods are unacceptable in practice.

[0142] Based on the above problems, an embodiment of the present disclosure provides an association annotation method for determining the associated persons of non-motor vehicles in a video.

[0143] The association annotation method provided by the embodiment of the present disclosure mainly includes: 1) video preprocessing, 2) annotation data preparation, 3) data annotation, and 4) annotation result processing, a total of four parts.

[0144] See also Figure 5 , Figure 5 This is an optional flow chart of video preprocessing provided by the embodiment of the present disclosure. Figure 5 The steps shown are explained.

[0145] S501: Analyze the video to be processed in the elevator scene to obtain an analysis result.

[0146] The video to be processed is captured by a video capture device in an elevator scene. The video capture device can be located inside the elevator and used to capture image data corresponding to real objects such as people and non-motorized vehicles within the elevator. In some embodiments, the image data can exist in the form of a video stream. For example, the image data can include multiple frames of scene images arranged in a time sequence. The video to be processed can be image data collected over a period of time.

[0147] In some embodiments, a human body detection and tracking algorithm may be used to analyze the video to be processed to obtain at least one tracking segment.

[0148] Specifically, for the multiple frames of scene images in the video to be processed, a detection result in each frame of the scene image can be determined based on a human body detection algorithm. The detection result includes at least detection frame information corresponding to the human object, and the detection frame information is used to determine the detection frame size and position of the human object in the scene image. In the case where the scene image includes multiple human objects, the scene image can include detection frame information corresponding to each human object. Subsequently, combining the temporal relationship between the scene images and the detection result in each scene image, at least one trajectory segment can be determined through a tracking algorithm.

[0149] Wherein, one trajectory segment corresponds to multiple frames of scene images. In the following embodiments, the number of frames of scene images corresponding to one trajectory segment is used as the length of the trajectory segment.

[0150] S502: For each tracking segment obtained through analysis, save multiple person images corresponding to the tracking segment.

[0151] In some embodiments, for each scene image in the tracking segment, a person image corresponding to the target person object can be captured from the scene image based on the detection result corresponding to the scene image, that is, based on the detection frame information corresponding to the target person object in the scene image. It should be noted that for a tracking segment, multiple scene images in the tracking segment all include the target person object, that is, the tracking segment is used to describe a segment of the trajectory of the target person object.

[0152] After obtaining the person images corresponding to each scene in the tracking segment, some person images may be filtered from all the person images to obtain multiple person images, and the filtered person images may be saved. The number of person images may be N, and based on the number N of person images, uniform sampling is performed from all the person images in the tracking segment to obtain N person images. N may be set to 5.

[0153] In other embodiments, based on the number N of character images, N scene images can be selected from all scene images corresponding to the tracking segment, and then based on the detection frame information corresponding to the target character object in the N scene images, the character image corresponding to the target character object can be intercepted to obtain N character images.

[0154] It should be noted that when generating a person image corresponding to each scene image, the person image may carry tag information, including frame information, trajectory information, and detection frame information. The frame information is the frame number of the scene image corresponding to the person image in the video to be processed; the trajectory information is the unique identifier of the trajectory segment in which the scene image corresponding to the person image resides; and the detection frame information is the size of the detection frame of the person object and its position in the scene image.

[0155] See also Figure 6 , which shows a schematic diagram of character images corresponding to two trajectory segments. It can be seen that the multiple character images corresponding to trajectory segment 61 are all character A; the multiple character images corresponding to trajectory segment 62 are all character B.

[0156] S503 : Splicing the character images corresponding to each trajectory segment to obtain a spliced ​​image corresponding to each trajectory segment.

[0157] In this embodiment, for each trajectory segment, the character images corresponding to the trajectory segment are spliced ​​to obtain a spliced ​​image corresponding to the trajectory segment.

[0158] See also Figure 6 , which shows a schematic diagram of the stitched images corresponding to two trajectory segments. It can be seen that the stitched images 63 corresponding to the trajectory segment 61 are all images of person A; the stitched images 64 corresponding to the trajectory segment 62 are all images of person B.

[0159] See also Figure 7 , Figure 7 This is an optional flow chart of the preparation of annotation data provided by the embodiment of the present disclosure. Figure 7 The steps shown are explained.

[0160] S701: Determine a reference image in the video to be processed, and obtain a labeling result corresponding to the reference image; the labeling result is used to determine a target person object associated with an illegal object in the video to be processed;

[0161] S702: Store the reference image carrying the annotation result and the spliced ​​image corresponding to each trajectory segment in a folder to be annotated.

[0162] See also Figure 8 , Figure 8 This is an optional flow chart of data annotation provided by the embodiment of the present disclosure. Figure 8 The steps shown are explained.

[0163] S801: Obtain a second annotation result for the spliced ​​image corresponding to each of the trajectory segments in the to-be-annotated folder; the second annotation result is used to determine whether the person object in the spliced ​​image corresponding to the trajectory segment is the same as the target person object.

[0164] In some embodiments, the reference image and each stitched image in the to-be-annotated folder may be compared manually, and a corresponding second annotation result may be added to each stitched image.

[0165] For any stitched image, it is possible to manually determine whether the target person in the reference image is identical to the person object in each person image in the stitched image, and based on the determination, add a corresponding second annotation result to each stitched image. For example, if the stitched image includes five person images, it is possible to determine whether the person appearing in the reference image is identical to the person appearing in each person image. If all five person images are identical, a second annotation result is added to the stitched image to indicate that the person object in the stitched image is identical to the target person object; if at least one person image among the five person images is inconsistent, a second annotation result is added to the stitched image to indicate that the person object in the stitched image is different from the target person object.

[0166] In some embodiments, a first feature map corresponding to the reference image can be extracted through a pre-trained feature extraction network; a second feature map corresponding to each character image in the spliced ​​image can be extracted through the feature extraction network, and the multiple second feature maps obtained are fused to obtain a third feature map; a similarity quantization value between the first feature map and the third feature map is obtained, and when the similarity quantization value exceeds a preset similarity threshold, a second annotation result is added to the spliced ​​image, indicating that the character object in the spliced ​​image is the same as the target character object; when the similarity quantization value does not exceed the preset similarity threshold, a second annotation result is added to the spliced ​​image, indicating that the character object in the spliced ​​image is different from the target character object.

[0167] In some embodiments, a first feature map corresponding to the reference image can also be extracted through a pre-trained feature extraction network; a second feature map corresponding to each character image in the stitched image can be extracted through the feature extraction network; similarity quantization values ​​between the first feature map and each of the second feature maps are respectively obtained, and when the obtained similarity quantization values ​​all exceed a preset similarity threshold, a second annotation result is added to the stitched image, indicating that the character object in the stitched image is the same as the target character object; when there is at least one similarity quantization value that does not exceed the preset similarity threshold, a second annotation result is added to the stitched image, indicating that the character object in the stitched image is different from the target character object.

[0168] S802 : For each of the stitched images corresponding to the trajectory segments, obtain a verification result of the second annotation result to obtain a third annotation result of the stitched image corresponding to each of the trajectory segments.

[0169] In some embodiments, for the stitched image corresponding to each trajectory segment, the stitched image and the corresponding second annotation result can be manually verified, and the corresponding verification result can be obtained. If the verification result indicates that the second annotation result is correct, the second annotation result can be used as the third annotation result. If the verification result indicates that the second annotation result is incorrect, the second annotation result can be adjusted to obtain the third annotation result.

[0170] It should be noted that, for the same folder to be labeled, there may be multiple stitched images carrying the second labeling results indicating that the person object in the stitched image is the same as the target person object, or there may be no stitched image carrying the second labeling results indicating that the person object in the stitched image is the same as the target person object.

[0171] The second annotation result may be annotated in the form of a numerical value. For example, the second annotation result indicating that the human object in the stitched image is identical to the target human object may be set to a first value, and the second annotation result indicating that the human object in the stitched image is different from the target human object may be set to a second value. For example, the first value is "1" and the second value is "0."

[0172] See also Figure 9 , Figure 9 This is an optional flowchart of the labeling result processing provided by the embodiment of the present disclosure. Figure 9 The steps shown are explained.

[0173] S901: Obtain a target track number corresponding to a target stitched image in the to-be-annotated folder; the target stitched image carries a second annotation result indicating that a person object in the stitched image is identical to the target person object.

[0174] S902: Use the detection result of each frame of the scene image of the trajectory segment corresponding to the target trajectory number as the associated annotation result.

[0175] The association annotation method provided in the above embodiment can efficiently and cost-effectively annotate the association relationships between human bodies and non-motor vehicles in videos, reducing the annotation cost compared to traditional annotation methods. It should be noted that the above association annotation method can be applied not only to the association annotation between human bodies and non-motor vehicles, but can also be extended to any other pairwise association annotation tasks, and even to the association annotation of any number of objects.

[0176] Figure 10 A schematic diagram of the structure of an apparatus for determining an association relationship provided by an embodiment of the present disclosure is shown in FIG. Figure 10 As shown, the association relationship determination device 1000 includes:

[0177] The first acquisition module 1001 is configured to acquire a plurality of trajectory segments corresponding to a video to be processed and a spliced ​​image corresponding to each of the trajectory segments; the spliced ​​image includes image information of a plurality of image frames of a first person object corresponding to the trajectory segment in the video to be processed;

[0178] The second acquisition module 1002 is configured to obtain a first annotation result for the spliced ​​image corresponding to each of the trajectory segments; the first annotation result is used to indicate whether the first and second human objects corresponding to the trajectory segments are the same; the second human object is a human object in the video to be processed that has an associated relationship with the target object;

[0179] The annotation module 1003 is configured to, for each of the trajectory segments, add a second annotation result to the multiple trajectory image frames corresponding to the trajectory segment based on the first annotation result of the stitched image corresponding to the trajectory segment, wherein the second annotation result is used at least to determine whether there is a human object associated with the target object in the multiple trajectory image frames corresponding to the trajectory segment.

[0180] In some embodiments, the first acquisition module 1001 is further configured to:

[0181] performing a first detection process and a tracking process on the video to be processed for a person object, to obtain a trajectory segment corresponding to each of a plurality of first person objects; the trajectory segment including a plurality of trajectory image frames and position information of the first person object in each of the trajectory image frames;

[0182] For each of the trajectory segments, a spliced ​​image corresponding to the trajectory segment is generated based on a plurality of trajectory image frames corresponding to the trajectory segment and position information of the first human object in each of the trajectory image frames.

[0183] In some embodiments, the first acquisition module 1001 is further configured to:

[0184] Screening the multiple trajectory image frames to obtain multiple image frames to be spliced;

[0185] generating a first person image corresponding to each of the image frames to be stitched based on the plurality of image frames to be stitched and position information of the first person object in each of the image frames to be stitched;

[0186] The spliced ​​image is generated based on the first person image corresponding to each of the image frames to be spliced.

[0187] In some embodiments, the first acquisition module 1001 is further configured to:

[0188] Performing a first detection process on the video to be processed for human objects to obtain a first detection result for each scene image frame in the video to be processed; the first detection result includes position information corresponding to each first human object in the scene image frame;

[0189] Performing a first detection process on the plurality of scene image frames for human objects to obtain a first detection result for each of the scene image frames, wherein the first detection result includes position information corresponding to each of the first human objects in the scene image frames;

[0190] Based on the frame number of each of the scene image frames and the position information corresponding to each of the first human objects, a trajectory segment corresponding to each of the first human objects is generated.

[0191] In some embodiments, the first acquisition module 1001 is further configured to:

[0192] For each of the trajectory segments, based on the spliced ​​image corresponding to the trajectory segment, obtaining a first person feature of a first person object corresponding to the trajectory segment;

[0193] Acquire a second person feature of the second person object in a reference image; the reference image is an image frame in the video to be processed that represents an association relationship between the second person object and the target object;

[0194] The first labeling result is generated based on the first character feature and the second character feature.

[0195] In some embodiments, the second acquisition module 1002 is further configured to:

[0196] For each of the trajectory segments, a stitched image and a reference image corresponding to the trajectory segment are displayed through a first interface; the reference image is an image frame in the video to be processed that represents an association relationship between the second person object and the target object;

[0197] A first annotation operation on the stitched image is received to obtain a first annotation result.

[0198] In some embodiments, the second acquisition module 1002 is further configured to:

[0199] Displaying a plurality of candidate reference images through a second interface; receiving a selection operation for a target candidate reference image among the plurality of candidate reference images, and determining the target candidate reference image as the reference image; or,

[0200] Obtain a frame number corresponding to each of the multiple candidate reference images; determine a target frame number based on the frame number corresponding to each of the candidate reference images, and determine the candidate reference image corresponding to the target frame number as the reference image.

[0201] In some embodiments, the video to be processed includes multiple scene image frames, and the second acquisition module 1002 is further configured to:

[0202] Determine the multiple scene image frames as the multiple reference images to be selected; or,

[0203] performing a second detection process on the plurality of scene image frames for the target object, and using the scene image frames including the target object as the plurality of reference images to be selected; or

[0204] performing a second detection process on the plurality of scene image frames for the target object to obtain at least one scene image frame including the target object and a second detection result corresponding to each of the scene image frames, and determining the plurality of candidate reference images based on the second detection result corresponding to each of the scene image frames; or

[0205] Perform a second detection process on the multiple scene image frames for the target object to obtain a second detection result corresponding to each of the scene image frames, wherein the second detection result includes position information of the target object in the scene image frame; perform a first detection process on the multiple scene image frames for the human object to obtain a first detection result for each of the scene image frames, wherein the first detection result includes position information corresponding to each of the first human objects in the scene image frame; determine the multiple reference images to be selected from the multiple scene image frames based on the first detection result and the second detection result corresponding to each of the scene image frames.

[0206] In some embodiments, the labeling module 1003 is further configured to:

[0207] When the first labeling result indicates that the first person object corresponding to the trajectory segment and the second person object are the same, obtaining a plurality of trajectory image frames corresponding to the trajectory segment;

[0208] Add the second annotation results to the multiple trajectory image frames; the second annotation results include at least one of the following: the first annotation result, position information of the first person object corresponding to the trajectory segment in each of the trajectory image frames, the frame number of each of the trajectory image frames, and the trajectory identifier corresponding to the trajectory segment.

[0209] In some embodiments, the annotation module 1003 is used to:

[0210] For each first annotation result of the spliced ​​image corresponding to the trajectory segment, a verification result of the first annotation result is obtained, and the first annotation result is adjusted based on the verification result.

[0211] The description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of the present disclosure, please refer to the description of the method embodiment of the present disclosure for understanding.

[0212] It should be noted that, in the embodiments of the present disclosure, if the above-mentioned method for determining the association relationship is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present disclosure is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a device to execute all or part of the methods of each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk. In this way, the embodiments of the present disclosure are not limited to any target combination of hardware and software.

[0213] Figure 11 A hardware entity diagram of an association relationship determination device provided in an embodiment of the present disclosure, such as Figure 11 As shown, the hardware entity of the association relationship determination device 1100 includes: a processor 1101 and a memory 1102, wherein the memory 1102 stores a computer program that can be run on the processor 1101, and the processor 1101 implements the steps in the method of any of the above embodiments when executing the program.

[0214] The memory 1102 stores computer programs that can be run on the processor. The memory 1102 is configured to store instructions and applications executable by the processor 1101. It can also cache data to be processed or processed by the processor 1101 and the modules in the association relationship determination device 1100 (for example, image data, audio data, voice communication data, and video communication data). It can be implemented through flash memory (FLASH) or random access memory (RAM).

[0215] When the processor 1101 executes the program, the steps of any of the above-mentioned methods for determining an association relationship are implemented. The processor 1101 generally controls the overall operation of the apparatus 1100 for determining an association relationship.

[0216] An embodiment of the present disclosure provides a computer storage medium storing one or more programs. The one or more programs can be executed by one or more processors to implement the steps of the association relationship determination method of any of the above embodiments.

[0217] It should be noted that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of the present disclosure, please refer to the description of the method embodiments of the present disclosure for understanding.

[0218] The processor may be at least one of an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a central processing unit (CPU), a controller, a microcontroller, and a microprocessor. It is understood that the electronic device that implements the functions of the processor may also be other electronic devices, which are not specifically limited in the embodiments of the present disclosure.

[0219] The above-mentioned computer storage medium / memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory (Flash Memory), a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); it can also be various terminals including one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc.

[0220] It should be understood that “one embodiment” or “an embodiment” or “an embodiment of the present disclosure” or “the aforementioned embodiments” or “some embodiments” mentioned throughout the specification means that the target features, structures or characteristics related to the embodiment are included in at least one embodiment of the present disclosure. Therefore, “in one embodiment” or “in an embodiment” or “an embodiment of the present disclosure” or “the aforementioned embodiments” or “some embodiments” appearing throughout the specification do not necessarily refer to the same embodiment. In addition, the features, structures or characteristics of these targets may be combined in one or more embodiments in any suitable manner. It should be understood that in the various embodiments of the present disclosure, the size of the serial numbers of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present disclosure. The serial numbers of the embodiments of the present disclosure mentioned above are for description only and do not represent the advantages and disadvantages of the embodiments.

[0221] Unless otherwise specified, the association relationship determination device executes any step in the embodiments of the present disclosure, and the processor of the association relationship determination device may execute the step. Unless otherwise specified, the embodiments of the present disclosure do not limit the order in which the association relationship determination device executes the following steps. In addition, the methods used to process data in different embodiments may be the same method or different methods. It should also be noted that any step in the embodiments of the present disclosure can be independently executed by the association relationship determination device, that is, when the association relationship determination device executes any step in the above embodiments, it can be independent of the execution of other steps.

[0222] In the several embodiments provided in the present disclosure, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0223] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed across multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the scheme of this embodiment.

[0224] In addition, all functional units in the embodiments of the present disclosure may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0225] The methods disclosed in the several method embodiments provided in this disclosure can be arbitrarily combined without conflict to obtain new method embodiments.

[0226] The features disclosed in the several product embodiments provided in this disclosure can be arbitrarily combined without conflict to obtain new product embodiments.

[0227] The features disclosed in several method or device embodiments provided in this disclosure may be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0228] Those skilled in the art will understand that all or part of the steps of implementing the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, and other media that can store program codes.

[0229] Alternatively, if the above-mentioned integrated unit of the present disclosure is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present disclosure can essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, an association relationship determination device, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks or optical disks.

[0230] In the embodiments of the present disclosure, descriptions of the same steps and the same contents in different embodiments can refer to each other. In the embodiments of the present disclosure, the term "and" does not affect the order of the steps.

[0231] The above description is merely an embodiment of the present disclosure, but the scope of protection of the present disclosure is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this disclosure should be included in the scope of protection of the present disclosure. Therefore, the scope of protection of the present disclosure should be based on the scope of protection of the claims.

Claims

1. A method for determining an association relationship, characterized in that: The method comprises: Acquire multiple trajectory segments corresponding to the video to be processed and a spliced ​​image corresponding to each of the trajectory segments; the spliced ​​image includes image information of multiple image frames of the first person object corresponding to the trajectory segment in the video to be processed; Obtaining a first annotation result for the stitched image corresponding to each of the trajectory segments; the first annotation result is used to indicate whether a first person object and a second person object corresponding to the trajectory segment are the same; the second person object is a person object associated with the target object in the video to be processed; For each of the trajectory segments, based on the first annotation result of the stitched image corresponding to the trajectory segment, adding a second annotation result to the multiple trajectory image frames corresponding to the trajectory segment, wherein the second annotation result is at least used to determine whether there is a human object associated with the target object in the multiple trajectory image frames corresponding to the trajectory segment; The obtaining of a first annotation result of the spliced ​​image corresponding to each of the trajectory segments includes: For each of the trajectory segments, a stitched image and a reference image corresponding to the trajectory segment are displayed through a first interface; the reference image is an image frame in the video to be processed that represents the association relationship between the second person object and the target object; a first annotation operation is received for the stitched image to obtain the first annotation result.

2. The method according to claim 1, characterized in that The step of obtaining a plurality of trajectory segments corresponding to the video to be processed and a spliced ​​image corresponding to each of the trajectory segments includes: performing a first detection process and a tracking process on the video to be processed for a person object, to obtain a trajectory segment corresponding to each of a plurality of first person objects; the trajectory segment including a plurality of trajectory image frames and position information of the first person object in each of the trajectory image frames; For each of the trajectory segments, a spliced ​​image corresponding to the trajectory segment is generated based on a plurality of trajectory image frames corresponding to the trajectory segment and position information of the first human object in each of the trajectory image frames.

3. The method according to claim 2, characterized in that The step of generating a spliced ​​image corresponding to the trajectory segment based on a plurality of trajectory image frames corresponding to the trajectory segment and position information of the first person object in each of the trajectory image frames includes: Screening the multiple trajectory image frames to obtain multiple image frames to be spliced; generating a first person image corresponding to each of the image frames to be stitched based on the plurality of image frames to be stitched and position information of the first person object in each of the image frames to be stitched; The spliced ​​image is generated based on the first person image corresponding to each of the image frames to be spliced.

4. The method according to claim 2 or 3, characterized in that The performing first detection and tracking processing on the video to be processed for a person object to obtain at least one trajectory segment corresponding to the first person object includes: Performing a first detection process on the video to be processed for human objects to obtain a first detection result for each scene image frame in the video to be processed; the first detection result includes position information corresponding to each first human object in the scene image frame; Performing a first detection process on the plurality of scene image frames for human objects to obtain a first detection result for each of the scene image frames, wherein the first detection result includes position information corresponding to each of the first human objects in the scene image frames; Based on the frame number of each of the scene image frames and the position information corresponding to each of the first human objects, a trajectory segment corresponding to each of the first human objects is generated.

5. The method according to claim 1, wherein The obtaining of a first annotation result of the spliced ​​image corresponding to each of the trajectory segments includes: For each of the trajectory segments, based on the spliced ​​image corresponding to the trajectory segment, obtaining a first person feature of a first person object corresponding to the trajectory segment; Acquire a second person feature of the second person object in a reference image; the reference image is an image frame in the video to be processed that represents an association relationship between the second person object and the target object; The first labeling result is generated based on the first character feature and the second character feature.

6. The method according to claim 5, characterized in that The method for obtaining the reference image includes at least one of the following: displaying a plurality of candidate reference images through a second interface; receiving a selection operation for a target candidate reference image among the plurality of candidate reference images, and determining the target candidate reference image as the reference image; Obtaining a frame number corresponding to each of the plurality of reference images to be selected; A target frame number is determined based on the frame number corresponding to each of the candidate reference images, and the candidate reference image corresponding to the target frame number is determined as the reference image.

7. The method according to claim 6, characterized in that The video to be processed includes a plurality of scene image frames, and the plurality of reference images to be selected are obtained in a manner including at least one of the following: Determining the multiple scene image frames as the multiple reference images to be selected; performing a second detection process on the plurality of scene image frames for the target object, and using the scene image frames including the target object as the plurality of reference images to be selected; performing a second detection process on the plurality of scene image frames for the target object to obtain at least one scene image frame including the target object and a second detection result corresponding to each of the scene image frames, and determining the plurality of candidate reference images based on the second detection result corresponding to each of the scene image frames; performing a second detection process on the plurality of scene image frames for the target object to obtain a second detection result corresponding to each of the scene image frames, wherein the second detection result includes position information of the target object in the scene image frame; Performing a first detection process on the plurality of scene image frames for human objects to obtain a first detection result for each of the scene image frames, wherein the first detection result includes position information corresponding to each of the first human objects in the scene image frames; The multiple reference images to be selected are determined from the multiple scene image frames based on the first detection result and the second detection result corresponding to each of the scene image frames.

8. The method according to claim 1, characterized in that The adding a second annotation result to the plurality of trajectory image frames corresponding to the trajectory segment based on the first annotation result of the spliced ​​image corresponding to the trajectory segment includes: When the first labeling result indicates that the first person object corresponding to the trajectory segment and the second person object are the same, obtaining a plurality of trajectory image frames corresponding to the trajectory segment; Add the second annotation results to the multiple trajectory image frames; the second annotation results include at least one of the following: the first annotation result, position information of the first person object corresponding to the trajectory segment in each of the trajectory image frames, the frame number of each of the trajectory image frames, and the trajectory identifier corresponding to the trajectory segment.

9. The method according to claim 1, characterized in that The obtaining of a first annotation result of the spliced ​​image corresponding to each of the trajectory segments includes: For each first annotation result of the spliced ​​image corresponding to the trajectory segment, a verification result of the first annotation result is obtained, and the first annotation result is adjusted based on the verification result.

10. A device for determining an association relationship, characterized in that: include: A first acquisition module is configured to acquire a plurality of trajectory segments corresponding to a video to be processed and a spliced ​​image corresponding to each of the trajectory segments; the spliced ​​image includes image information of a plurality of image frames of a first person object corresponding to the trajectory segment in the video to be processed; A second acquisition module is used to obtain a first annotation result of the spliced ​​image corresponding to each of the trajectory segments; The first annotation result is used to indicate whether the first and second person objects corresponding to the trajectory segment are the same; the second person object is a person object in the video to be processed that has an associated relationship with the target object; The obtaining of a first annotation result for the stitched image corresponding to each of the trajectory segments includes: for each of the trajectory segments, displaying the stitched image and a reference image corresponding to the trajectory segment through a first interface; the reference image is an image frame in the video to be processed that represents an association relationship between the second person object and the target object; receiving a first annotation operation for the stitched image, and obtaining the first annotation result; The annotation module is configured to, for each of the trajectory segments, add a second annotation result to the multiple trajectory image frames corresponding to the trajectory segment based on the first annotation result of the stitched image corresponding to the trajectory segment, wherein the second annotation result is at least used to determine whether there is a human object associated with the target object in the multiple trajectory image frames corresponding to the trajectory segment.

11. A device for determining an association relationship, characterized in that: include: memory and processor, The memory stores a computer program executable on the processor. When the processor executes the computer program, the steps of the method according to any one of claims 1 to 9 are implemented.

12. A computer storage medium, characterized in that The computer storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in any one of claims 1 to 9.

Images