Machine learning model training, video processing method and apparatus, device, and medium

Abstract

The present disclosure relates to a machine learning model training method and device, a video processing method and device, equipment and a medium. The present disclosure trains a machine learning model unsupervisedly, so that more unlabeled video resources can participate in the unsupervised training process without labeling each frame of image in a sample video, and the utilization rate of video resources is improved. In addition, the present embodiment determines the historical motion trajectory of each target object in the current image according to the identifier corresponding to at least one target object in the current image and the identifier corresponding to the target object in at least one historical image before the current image, and determines the mapping image of the current image according to the historical motion trajectory of each target object in the current image, so that the mapping image can reflect the real motion of the target object, and the precision of the machine learning model can be improved by using the high-quality mapping image for training.

Description

Technical Field

[0001] This disclosure relates to the field of information technology, and in particular to a machine learning model training, video processing method, apparatus, device, and medium. Background Technology

[0002] With the continuous development of shooting equipment, images or videos can be captured at any time. Because video presents dynamic images, the content presented in each frame of the same video is constantly changing compared to the previous frame, from the first frame to the last. For example, the target object presented in each frame may be different, or the same target object may be located in different positions in different images, and so on.

[0003] To track the same target object across multiple consecutive frames, existing techniques employ supervised learning algorithms to train a machine learning model, which is then used for tracking. However, supervised learning algorithms require manual annotation of the location of different target objects in each frame of the video, resulting in the underutilization of a large amount of unannotated video resources. Furthermore, since manually annotated images are limited, the machine learning model trained using supervised learning algorithms may overfit, leading to lower accuracy. Summary of the Invention

[0004] To address, or at least partially address, the aforementioned technical problems, this disclosure provides a machine learning model training method, video processing method, apparatus, device, and medium to improve the accuracy of machine learning models.

[0005] In a first aspect, embodiments of this disclosure provide a machine learning model training method, including:

[0006] For each frame of the sample video, at least one target region in the image is encoded by the machine learning model to be trained, so as to obtain the representation vector corresponding to at least one target object in the image. The target region includes one or more target objects.

[0007] Based on the representation vectors corresponding to at least one target object in the image, determine the identifiers corresponding to at least one target object in the image;

[0008] The current image is selected from the sample video. Based on the identifiers corresponding to at least one target object in the current image and the identifiers corresponding to target objects in at least one historical image before the current image, the historical motion trajectory of each target object in the current image is determined. Based on the historical motion trajectory of each target object in the current image, the mapping image of the current image is determined. The mapping image of the current image includes a first positive sample and a first negative sample corresponding to each target object in the current image.

[0009] The machine learning model to be trained is unsupervised by traversing each target object in the current image and at least based on the similarity between the target object and the first positive sample, and the similarity between the target object and the first negative sample.

[0010] Secondly, embodiments of this disclosure provide a video processing method, the method comprising:

[0011] Acquire a target video, which includes multiple frames of images;

[0012] For each frame of the multi-frame images, at least one target region in the image is encoded by a machine learning model to obtain a representation vector corresponding to at least one target object in the image. The target region includes one or more target objects. The machine learning model is trained by the method described in the first aspect.

[0013] Based on the representation vectors corresponding to at least one target object in the image, determine the identifiers corresponding to at least one target object in the image;

[0014] Based on the identifiers corresponding to at least one target object in the image and the identifiers corresponding to at least one target object in previous historical images, the uncertainty of the identifier corresponding to each target object in the image is calculated. The uncertainty is related to the similarity between target objects with the same identifier and the similarity between target objects with different identifiers.

[0015] Correct the identifiers in the image whose uncertainty is greater than or equal to a preset threshold.

[0016] Thirdly, embodiments of this disclosure provide a video processing method, the method comprising:

[0017] Acquire vehicle video, which includes multiple frames of images;

[0018] For each frame of the multi-frame images, at least one target region in the image is encoded by a machine learning model to obtain a representation vector corresponding to at least one vehicle in the image. The target region includes one or more vehicles. The machine learning model is trained by the method described in the first aspect.

[0019] Based on the representation vectors corresponding to at least one vehicle in the image, determine the identifiers corresponding to at least one vehicle in the image;

[0020] Based on the identifiers corresponding to at least one vehicle in the image and the identifiers corresponding to at least one vehicle in previous historical images, the uncertainty of the identifier corresponding to each vehicle in the image is calculated. The uncertainty is related to the similarity between vehicles with the same identifier and the similarity between vehicles with different identifiers.

[0021] Correct the identifiers in the image whose uncertainty is greater than or equal to a preset threshold.

[0022] Fourthly, embodiments of this disclosure provide a video processing method, the method comprising:

[0023] Acquire a video of a person, the video of which includes multiple frames of images;

[0024] For each frame of the multi-frame images, at least one target region in the image is encoded by a machine learning model to obtain a representation vector corresponding to at least one person in the image. The target region includes one or more people. The machine learning model is trained by the method described in the first aspect.

[0025] Based on the representation vectors corresponding to at least one person in the image, determine the identifiers corresponding to at least one person in the image;

[0026] Based on the identifiers corresponding to at least one person in the image and the identifiers corresponding to at least one person in previous historical images, the uncertainty of the identifier corresponding to each person in the image is calculated. The uncertainty is related to the similarity between people with the same identifier and the similarity between people with different identifiers.

[0027] Correct the identifiers in the image whose uncertainty is greater than or equal to a preset threshold.

[0028] Fifthly, embodiments of this disclosure provide a machine learning model training apparatus, comprising:

[0029] The encoding module is used to encode at least one target region in each frame of the sample video using a machine learning model to be trained, so as to obtain a representation vector corresponding to at least one target object in the image, wherein the target region includes one or more target objects.

[0030] The first determining module is used to determine the identifiers corresponding to at least one target object in the image based on the representation vectors corresponding to at least one target object in the image.

[0031] The second determining module is used to select the current image from the sample video, determine the historical motion trajectory of each target object in the current image based on the identifiers corresponding to at least one target object in the current image and the identifiers corresponding to target objects in at least one historical image before the current image, and determine the mapping image of the current image based on the historical motion trajectory of each target object in the current image. The mapping image of the current image includes a first positive sample and a first negative sample corresponding to each target object in the current image.

[0032] The training module is used to traverse each target object in the current image and perform unsupervised training on the machine learning model to be trained, based at least on the similarity between the target object and the first positive sample, and the similarity between the target object and the first negative sample.

[0033] Sixthly, embodiments of this disclosure provide an electronic device, including:

[0034] Memory;

[0035] Processor; and

[0036] Computer programs;

[0037] The computer program is stored in the memory and configured to be executed by the processor to implement the methods described in the first, second, third, or fourth aspects.

[0038] In a seventh aspect, embodiments of this disclosure provide a computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the methods described in the first aspect, the second aspect, the third aspect, or the fourth aspect.

[0039] The machine learning model training, video processing method, apparatus, device, and medium provided in this disclosure acquire a current image from a sample video and acquires a first positive sample and a first negative sample corresponding to each target object in the current image from a mapped image of the current image. Further, by traversing each target object in the current image, unsupervised training is performed on the machine learning model to be trained based at least on the similarity between the target object and the first positive sample, and the similarity between the target object and the first negative sample. This eliminates the need to label each frame of the sample video, allowing more unlabeled video resources to participate in the unsupervised training process and improving the utilization rate of video resources. Additionally, this embodiment determines the historical motion trajectory of each target object in the current image based on the identifier corresponding to at least one target object in the current image and the identifier corresponding to the target object in at least one historical image preceding the current image. Based on the historical motion trajectory of each target object in the current image, a mapped image of the current image is determined, enabling the mapped image to reflect the true motion of the target object. Then, by performing unsupervised training on the machine learning model to be trained based on the first positive sample and the first negative sample in the high-quality mapped image, the accuracy of the machine learning model can be improved. This allows unsupervised training methods to achieve accuracy that meets or even surpasses that of supervised training methods. Attached Figure Description

[0040] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0041] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 A flowchart of a machine learning model training method provided in this embodiment of the disclosure;

[0043] Figure 2 A schematic diagram illustrating an application scenario provided by an embodiment of this disclosure;

[0044] Figure 3 A schematic diagram of a sample video provided for an embodiment of this disclosure;

[0045] Figure 4 A flowchart of a machine learning model training method provided in this embodiment of the disclosure;

[0046] Figure 5 A schematic diagram illustrating the determination of a target object identifier according to another embodiment of this disclosure;

[0047] Figure 6 A schematic diagram of a correction mark provided in another embodiment of this disclosure;

[0048] Figure 7 A flowchart of a machine learning model training method provided in another embodiment of this disclosure;

[0049] Figure 8 A flowchart of a machine learning model training method provided in another embodiment of this disclosure;

[0050] Figure 9 A flowchart of a machine learning model training method provided in this embodiment of the disclosure;

[0051] Figure 10 This is a flowchart of a video processing method provided in another embodiment of the present disclosure;

[0052] Figure 11 This is a flowchart of a video processing method provided in another embodiment of the present disclosure;

[0053] Figure 12 This is a flowchart of a video processing method provided in another embodiment of the present disclosure;

[0054] Figure 13 This is a schematic diagram of the structure of a machine learning model training apparatus provided in an embodiment of the present disclosure;

[0055] Figure 14 This is a schematic diagram of the structure of the video processing apparatus provided in the embodiments of this disclosure;

[0056] Figure 15 A schematic diagram of the structure of an electronic device embodiment provided in this disclosure. Detailed Implementation

[0057] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0058] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0059] It should be noted that the sample videos or target videos (including but not limited to videos taken by user devices, videos pre-stored in cloud servers, etc.) and target objects in the images involved in this application (including but not limited to vehicles, people, houses, animals, etc.) are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0060] In addition, the machine learning model training and video processing methods provided in this application involve the following explanations of terms, detailed below:

[0061] Multi-object tracking (MOT) refers to determining the identity (ID) of each target object in each frame of a video stream and generating the motion trajectory, i.e., the change of position information of the same target object between different frames.

[0062] Target object identifier: A unique identifier for the target object.

[0063] Tracklet: A collection of positional information of the same target object in different frames.

[0064] Typically, images or videos can be captured at any time using a camera. Since video presents a dynamic picture, the content presented in each frame of the same video changes continuously from the first frame to the last. For example, the target object may differ in each frame, or the same target object may be located in different positions in different images. Because the target object is moving relative to the camera, and some images may contain multiple target objects with interference between them (e.g., mutual occlusion, similar appearance), existing technologies cannot accurately track the same target object across multiple consecutive frames. To address this problem, this disclosure provides a machine learning model training method, which will be described below with reference to specific embodiments.

[0065] Figure 1This is a flowchart illustrating a machine learning model training method provided in this embodiment. The method can be executed by a machine learning model training device, which can be implemented in software and / or hardware. This device can be configured in an electronic device, such as a server or terminal, where the terminal specifically includes a mobile phone, computer, or tablet computer. The server can specifically be a cloud server, and the machine learning model training method can be executed in the cloud. Several computing nodes (cloud servers) can be deployed in the cloud, each with computing, storage, and other processing resources. In the cloud, multiple computing nodes can be organized to provide a certain service; of course, a single computing node can also provide one or more services. The cloud can provide the service by providing a service interface, which users call to use the corresponding service. Service interfaces include Software Development Kits (SDKs), Application Programming Interfaces (APIs), etc. Furthermore, the machine learning model training method described in this embodiment is applicable to... Figure 2 The application scenarios shown are as follows. Figure 2 As shown, this application scenario includes a terminal 21 and a server 22. A machine learning model to be trained is deployed on the server 22. This machine learning model can be a machine learning model in general, or a portion of a layer or processing logic within a machine learning model. The server 22 can perform unsupervised training on the machine learning model deployed on the server 22 using the machine learning model training method described in this embodiment. After training is complete, the server 22 can retain the trained machine learning model locally, allowing the server 22 to process the target video using the trained machine learning model, thereby identifying at least one target object appearing in each frame of the target video. Alternatively, the server 22 can deploy the trained machine learning model to the terminal 21, allowing the terminal 21 to process the target video using the trained machine learning model, thereby identifying at least one target object appearing in each frame of the target video. The following is a combination of... Figure 2 This method will be described in detail, such as Figure 1 As shown, the specific steps of this method are as follows:

[0066] S101. For each frame of the sample video, at least one target region in the image is encoded by the machine learning model to be trained to obtain the representation vector corresponding to at least one target object in the image. The target region includes one or more target objects.

[0067] For example, terminal 21 has a shooting function, which can capture sample videos. Further, terminal 21 sends the sample videos to server 22. Alternatively, server 22 may have pre-stored sample videos. Or, server 22 may obtain the sample videos from other terminals or other servers. In other words, this embodiment does not limit the source of the sample videos.

[0068] Understandably, the sample video comprises multiple frames. When server 22 acquires the sample video, it can perform target recognition on each frame to identify at least one target object in each frame. This target object can be static or moving. For example, Figure 3 Image 31 is the first frame of the sample video. After performing target recognition on image 31, server 22 outputs the category and location of each target object in image 31. For example, the category of target object 11 is "vehicle," and the location of target object 11 is the position of the bounding box outside target object 11 in image 31. The category of target object 12 is "vehicle," and the location of target object 12 is the position of the bounding box outside target object 12 in image 31. Similarly, server 22 can perform target recognition on every frame of the sample video except the first frame, thereby identifying at least one target object in each frame after the first frame. For example, image 32 is the second frame of the sample video, and image 33 is the last frame of the sample video. The target objects identified by server 22 from each frame are as follows: Figure 3 As shown. It is understandable that if the target object is moving, the position of the same target object may be different in different images, and the number of target objects appearing in different images may be different. For example, image 33 not only shows target objects 11 and 12, but also a new target object 13. Furthermore, in this embodiment, the image region corresponding to the rectangular frame used to represent the position of the target object in each frame is recorded as the target region, such as... Figure 3 As shown, each target region includes one target object. In other embodiments, some target regions may also include multiple target objects.

[0069] Furthermore, server 22 can encode each target region in each frame of the image using the machine learning model to be trained, obtaining a representation vector corresponding to each target object in each frame of the image. Specifically, the machine learning model includes an encoder, and the encoding here can be implemented using an encoder.

[0070] For example, the target region corresponding to target object 11 in image 31 is input into the machine learning model to be trained, so that the machine learning model outputs the representation vector of the target region. Here, the representation vector of the target region can be used as the representation vector of target object 11, or the representation vector of target object 11 can be obtained after pre-processing the representation vector of the target region. That is, the representation vector of target object 11 is determined based on the representation vector of the target region. Similarly, the representation vector of target object 12 in image 31, and the representation vector of each target object in each frame of the image after image 31 can be obtained.

[0071] It is understandable that because the same target object occupies different positions in different images, the target region corresponding to the same target object is different in different images, resulting in different representation vectors for the same target object in different images. For example Figure 3 As shown, assuming the target object 11 is in motion, its position differs in different images. For example, the position of target object 11 in image 31 is different from its position in image 32. Consequently, the target region corresponding to target object 11 in image 31 is different from its position in image 32. These two target regions may differ in size, position, and the number of pixels they contain, resulting in different representation vectors for target object 11 in image 31 and image 32. However, the similarity between these two representation vectors will be greater than or equal to a preset similarity. Furthermore, for the same image, since different target objects occupy different positions within the image, their representation vectors will also be different. For example, target object 11 and target object 12 in image 31 are located in different positions; therefore, the representation vectors for target object 11 and target object 12 in image 31 will be different.

[0072] In addition, such as Figure 3 Each frame of the image shown includes multiple target objects. In some other embodiments, some images in the sample video may include target objects, while others may not. Furthermore, for the images that include target objects, some images may include one target object, while others may include multiple target objects. That is, for any given frame of the sample video, the image may or may not include target objects; if target objects are included, the image may include one or more, i.e., at least one, target object.

[0073] S102. Determine the identifiers corresponding to at least one target object in the image based on the representation vectors corresponding to at least one target object in the image.

[0074] In some scenarios, different target objects appearing in sample videos may be of the same or similar categories, resulting in similar appearances such as shape, color, and brightness. Therefore, in order to distinguish the target objects appearing in each frame, different unique identifiers can be assigned to the different target objects appearing in each frame.

[0075] For example Figure 3 As shown, the different target objects appearing in each frame of the image are all vehicles. Therefore, in order to distinguish the various target objects appearing in each frame of the image, the server 22 can determine the identifier of each target object in each frame of the image based on the representation vector of each target object included in each frame of the image.

[0076] Optionally, determining the identifiers corresponding to at least one target object in the image based on the representation vectors corresponding to at least one target object in the image includes: if the image is the first image in the sample video where a target object appears, then assigning a different identifier to each target object in the image based on the representation vectors corresponding to at least one target object in the image.

[0077] For example Figure 3 As shown, image 31 can also be the first image in the sample video where the target object appears. For example, in some embodiments, the target object already exists in the first frame of the sample video; in this case, image 31 can be that first frame. In other embodiments, the target object may not exist in the first N frames of the sample video, and the target object may only appear from the (N+1)th frame; in this case, image 31 can be that (N+1)th frame. Specifically, N is an integer greater than or equal to 1.

[0078] In image 31, since the representation vectors of target object 11 and target object 12 are different, server 22 determines that target object 11 and target object 12 are different target objects. Therefore, server 22 can assign a first identifier to target object 11 and a second identifier to target object 12. The first identifier and the second identifier are different. Specifically, the first identifier or the second identifier can be numbers, letters, strings, etc. That is to say, this embodiment does not limit the representation form of the identifiers corresponding to each target object, as long as it can distinguish each target object in each frame of the image. This embodiment uses numbers as an example; specifically, the identifiers of each target object in the same image can be consecutive numbers. For example... Figure 3 As shown, in image 31, the identifier assigned to target object 11 is 1, and the identifier assigned to target object 12 is 2.

[0079] Furthermore, based on the identifiers of each target object in image 31, the identifiers of each target object in image 32 can be determined. For example, the representation vector of target object 11 in image 31 is quite similar to its representation vector in image 32, and the representation vectors of target object 12 in image 31 and image 32 are also quite similar. Therefore, the identifier of target object 11 in image 32 is determined to be 1, and the identifier of target object 12 in image 32 is determined to be 2. Similarly, when a new target object 13 appears in image 32, since the representation vector of target object 13 in image 32 differs significantly from the representation vectors of other target objects, a new identifier, such as 3, can be assigned to target object 13.

[0080] S103. Select the current image from the sample video, determine the historical motion trajectory of each target object in the current image based on the identifiers corresponding to at least one target object in the current image and the identifiers corresponding to target objects in at least one historical image before the current image, and determine the mapping image of the current image based on the historical motion trajectory of each target object in the current image. The mapping image of the current image includes the first positive sample and the first negative sample corresponding to each target object in the current image.

[0081] Understandably, due to the imprecision of the machine learning model being trained, in some scenarios, the representation vectors of the same target object may differ significantly across different images, while the differences in representation vectors of different target objects may be small, leading to inaccurate identification of each target object. Therefore, after determining the identifier of at least one target object in each frame, the current image can be selected from the sample video. For example, assuming the sample video includes 50 frames, each containing a target object, the current image can be any frame in the sample video except the first frame. Specifically, if the current image is the second frame, then at least one previous historical image includes the first frame. If the current image is the third frame, then at least one previous historical image includes the first and second frames, and so on.

[0082] Furthermore, based on the identifiers corresponding to at least one target object in the current image and the identifiers corresponding to target objects in at least one historical image preceding the current image, the historical motion trajectory of each target object in the current image is determined.

[0083] For example, the current image is as follows Figure 3Image 33 is shown above, and several historical images precede image 33. Based on the identifiers corresponding to each target object in image 33, and the identifiers corresponding to each target object that appeared in the previous historical images, the historical motion trajectory of the target object with the same identifier from the first frame to the current image can be determined. For example, target object 11 in the current image 33 is identified by identifier 1, and target objects with identifier 1 also appeared in the previous historical images. Therefore, the historical motion trajectory of target object 11 from the first frame image 31 to the current image 33 can be determined. Similarly, the historical motion trajectories of other target objects in the current image 33, such as target object 12 and target object 13, can be determined.

[0084] Furthermore, based on the historical motion trajectories corresponding to target objects 11, 12, and 13 in the current image 33, a mapping image of the current image 33 is determined. For example, the mapping image of the current image 33 is as follows: Figure 3 Image 34 is shown.

[0085] It is understandable that, since the mapped image is an image obtained by mapping the current image, each target object appearing in the current image will also appear in the mapped image. For example, Figure 3 Image 34 is a mapping of image 33. Each target object in image 33 is mapped into image 34. For example, after mapping target object 12, identified as 2 in image 33, a target object 12 identified as 2 will also appear in image 34, but its position in image 33 will be different from its position in image 34. Therefore, the representation vector of target object 12 in image 33 will be different from its representation vector in image 34. Similarly, the representation vectors of other target objects in image 33 will also change after mapping.

[0086] Since images 33 and 34 are two different images, and the target objects appearing in image 33 also appear in image 34, for each target object in image 33, there are target objects in image 34 with the same identifier and target objects with different identifiers. For example, taking target object 11 in image 33 as an example, target object 11 in image 34 has the same identifier as target object 11 in image 33. Therefore, target object 11 in image 34 is recorded as the first positive sample of target object 11 in image 33. Target objects 12 and 13 in image 34 have different identifiers from target object 11 in image 33. Therefore, target objects 12 and 13 in image 34 are recorded as the first negative samples of target object 11 in image 33, respectively. Similarly, for target objects 12 and 13 in image 33, the first positive sample and first negative sample corresponding to target object 12, and the first positive sample and first negative sample corresponding to target object 13, can be determined from image 34, respectively.

[0087] S104. Traverse each target object in the current image, and perform unsupervised training on the machine learning model to be trained based at least on the similarity between the target object and the first positive sample, and the similarity between the target object and the first negative sample.

[0088] Understandably, although target object 11 in image 33 and target object 11 in image 34 correspond to the same identifier, the representation vectors of target object 11 in image 33 and target object 11 in image 34 are different. Furthermore, if the machine learning model is not accurate enough, the difference between the representation vectors of target object 11 in image 33 and target object 11 in image 34 may be large, resulting in a large distance between these two representation vectors and thus a low similarity between target object 11 in image 33 and target object 11 in image 34. Similarly, if the machine learning model is not accurate enough, the difference between the representation vectors of target object 11 in image 33 and target object 12 in image 34 may be small, resulting in a small distance between these two representation vectors and thus a high similarity between target object 11 in image 33 and target object 12 in image 34. Similarly, this leads to a high similarity between target object 11 in image 33 and target object 13 in image 34. In other words, when the machine learning model is not accurate enough, the similarity between target objects with the same label in images 33 and 34 will be low, while the similarity between target objects with different labels will be high. Ideally, the similarity between target objects with the same label should be as high as possible, while the similarity between target objects with different labels should be as low as possible. Therefore, this embodiment can traverse each target object in the current image and perform unsupervised training on the machine learning model to be trained, based at least on the similarity between the target object and the first positive sample, and the similarity between the target object and the first negative sample. This allows the trained machine learning model to increase the similarity between target objects with the same label and decrease the similarity between target objects with different labels. The similarity between the target object and the first positive sample can be the similarity between the representation vector of the target object in the current image and the representation vector of the first positive sample corresponding to the target object in the mapped image. Similarly, the similarity between the target object and the first negative sample can be the similarity between the representation vector of the target object in the current image and the representation vector of the first negative sample corresponding to the target object in the mapped image.

[0089] Furthermore, to improve the accuracy of the machine learning model, this embodiment can perform multiple rounds of iterative training. In each round of training, the machine learning model can undergo multiple iterations. Specifically, in each round of training, one iteration can be implemented according to S103 and S104. For example, according to S103, the current image is selected from the sample video, and after training the machine learning model once according to S104, S103 is executed again to change the current image, and the machine learning model is trained again according to S104. This process continues until every frame in the sample video, except for the first frame, has been used as the current image and participated in the training, at which point one round of training is considered complete. In the next round of training, the machine learning model trained in the previous round can be used to encode at least one target region in each frame of the sample video, and then the corresponding identifier for each target region in each frame can be determined. Furthermore, for each current image selected from the sample video, an iterative training is performed according to S103 and S104 until every frame in the sample video except the first frame has been used as the current image and participated in the training. This is considered the end of the next round of iterative training. This process continues until the preset number of iterative training rounds is reached, or the parameters of the machine learning model converge. At this point, training is considered complete, resulting in a trained machine learning model. Because the trained machine learning model has high accuracy, it can ensure that the similarity between target objects with the same label is sufficiently high, while simultaneously minimizing the similarity between target objects with different labels.

[0090] This embodiment obtains the current image from a sample video and acquires the first positive and first negative samples corresponding to each target object in the current image from the mapped image of the current image. Further, it iterates through each target object in the current image and performs unsupervised training on the machine learning model to be trained, based at least on the similarity between the target object and the first positive sample, and the similarity between the target object and the first negative sample. This eliminates the need to label every frame of the sample video, allowing more unlabeled video resources to participate in the unsupervised training process and improving the utilization rate of video resources. Additionally, this embodiment determines the historical motion trajectory of each target object in the current image based on the identifiers corresponding to at least one target object in the current image and the identifiers corresponding to target objects in at least one previous historical image. Based on the historical motion trajectories of each target object in the current image, it determines the mapped image of the current image, ensuring that the mapped image reflects the true motion of the target object. Then, by performing unsupervised training on the machine learning model to be trained using the first positive and first negative samples from the high-quality mapped image, the accuracy of the machine learning model can be improved. Thus, the accuracy of the unsupervised training method reaches or even surpasses the accuracy of the supervised training method.

[0091] In the above embodiments, several possible implementations were proposed to determine the identifiers corresponding to at least one target object in the image based on the representation vectors corresponding to at least one target object in the image. For example, in image 31, the identifier assigned to target object 11 is 1, and the identifier assigned to target object 12 is 2. Furthermore, based on the identifiers of each target object in image 31, the identifiers of each target object in image 32 can be determined. Another possible implementation is described below.

[0092] Optionally, based on the representation vectors corresponding to at least one target object in the image, an identifier corresponding to each target object in the image is determined, including, for example... Figure 4 The following steps are shown:

[0093] S401. If the image is not the first image in the sample video to show the target object, then determine the identifier corresponding to at least one target object in the image based on the representation vector corresponding to at least one target object in the image and the identifier corresponding to at least one target object in the historical images preceding the image.

[0094] For example Figure 5Image 51 shown is the first image in the sample video where the target objects appear. Image 51 includes target objects A, B, and C, and target objects A, B, and C are assigned labels 1, 2, and 3 respectively. Image 52 is not the first image in the sample video where the target objects appear; for example, image 52 is an image following image 51, and image 51 is a historical image of image 52. Specifically, images 51 and 52 may or may not be adjacent. Using the method described above, the representation vector corresponding to each target object in image 52 can be determined. Further, based on the representation vector corresponding to each target object in image 52 and the label corresponding to each target object in image 51, the label corresponding to each target object in image 52 is determined.

[0095] Optionally, determining the identifiers corresponding to at least one target object in the image based on the representation vectors corresponding to at least one target object in the image and the identifiers corresponding to at least one target object in previous historical images includes: determining the similarity between each target object in the image and each target object in the historical images based on the representation vectors corresponding to at least one target object in the image and the representation vectors corresponding to at least one target object in previous historical images; and determining the identifiers corresponding to at least one target object in the image based on the similarity between each target object in the image and each target object in the historical images and the identifiers corresponding to at least one target object in the historical images.

[0096] For example, based on the representation vectors corresponding to each target object in image 52 and each target object in image 51, the similarity between each target object in image 52 and each target object in image 51 is determined. The similarity between any target object in image 52 and any target object in image 51 can be the similarity between the representation vectors of any target object in image 52 and any target object in image 51. Because the machine learning model to be trained is not accurate enough, and target object B and target object C have relatively similar appearances, the representation vector of target object B in image 52 has the highest similarity with the representation vector of target object C in image 51, and the representation vector of target object C in image 52 has the highest similarity with the representation vector of target object B in image 51. Therefore, based on the identifier 3 corresponding to target object C in image 51, the identifier corresponding to target object B in image 52 is determined to be 3. Based on the identifier 2 corresponding to target object B in image 51, the identifier corresponding to target object C in image 52 is determined to be 2. Furthermore, since the appearance differences between target object A and target object B are significant, and the appearance differences between target object A and target object C are also significant, the representation vector of target object A in image 52 has the highest similarity to the representation vector of target object A in image 51. Therefore, the identifier corresponding to target object A in image 52 is determined to be 1.

[0097] S402. Based on the identifiers corresponding to at least one target object in the image and the identifiers corresponding to at least one target object in previous historical images, calculate the uncertainty of the identifier corresponding to each target object in the image. The uncertainty is related to the similarity between target objects with the same identifier and the similarity between target objects with different identifiers.

[0098] like Figure 5 As shown, when determining the identifiers of each target object in image 52 based on the identifiers of each target object in image 51, the identifiers of some target objects in image 52 may be inaccurate. Therefore, in this embodiment, after determining the identifiers of each target object in image 52 based on the identifiers of each target object in image 51, the uncertainty of the identifier corresponding to each target object in image 52 can be calculated based on the identifiers of each target object in image 52 and the identifiers of each target object in image 51. This uncertainty is related to the similarity between target objects with the same identifier and the similarity between target objects with different identifiers. Specifically, this uncertainty is inversely proportional to the similarity between target objects with the same identifier and directly proportional to the similarity between target objects with different identifiers.

[0099] Optionally, the uncertainty of the identifier corresponding to each target object in the image is calculated based on the identifiers corresponding to at least one target object in the image and the identifiers corresponding to at least one target object in the previous historical images. This includes: determining, based on the identifier corresponding to any target object in the image, a target object in the previous historical images that has the same identifier as the target object, and multiple target objects in the historical images that have different identifiers than the target object; and calculating the uncertainty of the identifier corresponding to any target object based on the similarity between the target object and any target object, and the maximum value among the similarity between the multiple target objects and any target object.

[0100] For example, any target object in image 52 is the i-th target object in image 52. Based on the identifier of this target object in image 52, a target object in image 51 with the same identifier as this target object is determined. For example, the j-th target object in image 51, that is, the i-th target object in image 52 and the j-th target object in image 51, have the same identifier. Here, the similarity between the i-th target object in image 52 and the j-th target object in image 51 is denoted as C. i,j In image 51, all target objects except the j-th target object have different identifiers from any given target object. Assume there are multiple target objects in image 51 with different identifiers from any given target object, and each of these multiple target objects has a similarity score with any given target object. Here, the maximum similarity score between these multiple target objects and any given target object is denoted as C. i,j2 Then the uncertainty δ of the identifier corresponding to any target object i It can be expressed as the following formula (1):

[0101] δ i =-logC i,j -log(1-C i,j2 (1)

[0102] From the above formula, we can see that δ i With C i,j Inversely proportional, δ i With C i,j2 They are directly proportional. Here, δ i This is denoted as object uncertainty.

[0103] Understandably, formula (1) uses C i,j With C i,j2These two factors are used to calculate the uncertainty of the target object. In other embodiments, the uncertainty of the target object can be calculated based on more factors. For example, a graph data structure is constructed based on each target object in image 52 and each target object in image 51, where each node in the graph represents a target object in image 52 or image 51, and the edge between two nodes represents the relationship between the two target objects, such as similarity. Based on this graph, the uncertainty of the identifier corresponding to each target object in image 52 is calculated. Specifically, the graph is processed using a graph neural network (GNN) to obtain the uncertainty of the identifier corresponding to each target object in image 52.

[0104] For example, taking target object A in image 52 as an example, since the identifier of target object A in image 52 is 1, a target object with the same identifier as target object A in image 52 can be identified from image 51 based on this identifier 1, such as target object A in image 51. Additionally, multiple target objects with different identifiers corresponding to target object A in image 52 can be identified from image 51, such as target object B and target object C in image 51. Further, the similarity between target object A in image 52 and target object A in image 51 is denoted as C. i,j Then, select the maximum similarity from the similarity between target object A in image 52 and target object B in image 51, and the maximum similarity between target object A in image 52 and target object C in image 51, and denote this maximum similarity as C. i,j2 The uncertainty δ of the identifier corresponding to target object A in image 52. i It can be obtained according to the above formula (1).

[0105] Similarly, according to the above formula (1), the uncertainty of the identifier corresponding to target object B in image 52 and the uncertainty of the identifier corresponding to target object C in image 52 can also be calculated.

[0106] S403. Correct the markings in the image whose uncertainty is greater than or equal to a preset threshold.

[0107] Due to δ i With C i,j Inversely proportional, δ i With C i,j2 The uncertainty is directly proportional to the threshold value. Therefore, when the uncertainty of the corresponding labels for target objects B and C in image 52 is greater than or equal to the preset threshold, it indicates that the corresponding labels for target objects B and C in image 52 are incorrect. At this point, the labels corresponding to target objects B and C in image 52 can be corrected, and the corrected result is as follows: Figure 6As shown. A specific correction method could be to swap the identifiers corresponding to target object B and target object C in image 52. Alternatively, the identifiers corresponding to target object B and target object C in image 52 could be redefined using the method described above.

[0108] It is understood that images 51 and 52 described in this embodiment are merely illustrative. The method described in this embodiment can calculate the uncertainty of the identifiers corresponding to each target object in other images of the sample video, and correct the identifiers corresponding to each target object in other images based on the uncertainty.

[0109] This embodiment calculates the uncertainty of the identifier corresponding to each target object in the image, making the uncertainty correlated with the similarity between target objects with the same identifier and with the similarity between target objects with different identifiers. Identifiers with uncertainties greater than or equal to a preset threshold in the image are corrected based on the uncertainty, thereby improving the accuracy of the identifier corresponding to each target object in the image. This ensures that the same target object corresponds to the same identifier in different images as much as possible. Therefore, the identifier of the same target object can maintain long-term consistency in the sample video, avoiding identification errors or switching between adjacent identifiers due to target object occlusion, camera shake, or other reasons.

[0110] The calculation process of the mapped image will be described in detail below with reference to specific embodiments.

[0111] Optionally, a mapped image of the current image is determined based on the historical motion trajectory of each target object in the current image, including, for example... Figure 7 The following steps are shown:

[0112] S701. Based on the historical motion trajectory of each target object in the current image, determine the target objects that meet the preset conditions from the current image. The target objects that meet the preset conditions have the highest probability of corresponding to the same identifier in their historical motion trajectory.

[0113] like Figure 3As shown, based on the target objects included in each frame of the sample video and the corresponding identifiers of the target objects, the historical motion trajectory of each target object in the current image, such as image 33, can be determined. For example, the historical motion trajectory of target object 11 from the first frame image 31 to the current image 33, the historical motion trajectory of target object 12 from the first frame image 31 to the current image 33, and the historical motion trajectory of target object 13 from the second frame image 32 to the current image 33. According to the method described in the above embodiment, the uncertainty of the identifier corresponding to target object 11 in each frame image starting from the second frame image can be calculated, as can the uncertainty of the identifier corresponding to target object 12 in each frame image starting from the second frame image, and the uncertainty of the identifier corresponding to target object 13 in each frame image starting from the second frame image. For the i-th target object in image 33, the cumulative uncertainty Ω of the i-th target object is calculated based on the uncertainty of the identifier corresponding to the i-th target object in each frame image starting from the second frame image. i Ω i It can be expressed as the following formula (2):

[0114]

[0115] in, t1 represents the uncertainty of the identifier corresponding to the i-th target object in the second frame image, and t1 represents the time corresponding to the second frame image in the sample video. Let t1 represent the uncertainty of the identifier corresponding to the i-th target object in the third frame image, and t2 represent the time corresponding to the third frame image in the sample video. And so on. Let represent the uncertainty of the identifier corresponding to the i-th target object in the current image, and t represent the time corresponding to the current image in the sample video. The corresponding calculation methods can be referred to formula (1) as described above. n represents... The number of [objects]. It is understandable that the i-th target object may appear in every frame from the second frame to the current frame, or it may appear in a subset of frames from the second frame to the current frame. Therefore, Specifically, Ω can represent the uncertainty of the identifier corresponding to the i-th target object in the images in which it appears, where n represents the number of images in which the i-th target object appears from the second frame to the current image. Here, Ω... i This is denoted as tracklet uncertainty.

[0116] According to formula (2), the small-segment trajectory uncertainty of each target object in the current image, such as image 33, can be calculated. Specifically, the small-segment trajectory uncertainty of different target objects is different. Therefore, based on the small-segment trajectory uncertainty of each target object in image 33, a target object that meets the preset conditions can be determined from image 33. The target object that meets the preset conditions has the smallest small-segment trajectory uncertainty. It can be understood that in formula (2), when When each of the elements is small, the calculated Ω i It is also relatively small. Therefore, for the target object that meets the preset conditions, since its short trajectory uncertainty is minimal, it can be determined that the uncertainty of the corresponding identifier of the target object that meets the preset conditions in each frame image starting from the second frame image is also relatively small, that is... Each of the values ​​in the image is relatively small. Therefore, the identifier corresponding to the target object that meets the preset conditions is the most stable during the process from the second frame image to the current image. That is, the probability that the target object that meets the preset conditions corresponds to the same identifier in its historical motion trajectory is the highest. For example Figure 3 In the current image 33, target objects 11, 12, and 13 correspond to Ω respectively. i And the Ω corresponding to target object 11 i The smallest value indicates that target object 11 has the highest probability of corresponding to the same identifier in its historical movement trajectory. Therefore, the historical movement trajectory of target object 11 is more accurate than that of target object 12 or target object 13.

[0117] S702. Based on the uncertainty of the corresponding identifier of the target object that meets the preset conditions in the at least one historical image, determine the target historical image from the at least one historical image, wherein the uncertainty of the corresponding identifier of the target object that meets the preset conditions in the target historical image is the largest.

[0118] For example Figure 3 The image 33 shown is preceded by multiple historical images. The target object, such as target object 11, that meets the preset conditions may appear in each of these historical images, or it may appear in some of them. Assuming that target object 11 appears in at least one historical image preceding image 33, the target historical image is determined from these at least one historical image based on the uncertainty of the identifier corresponding to target object 11 in that image, such that the uncertainty of the identifier corresponding to the target object meeting the preset conditions is maximized in the target historical image. For example, if the uncertainty of the identifier corresponding to target object 11 is maximized in the second frame image 32, then the second frame image 32 can be used as the target historical image.

[0119] S703. Determine the mapping image of the current image based on the current image and the target historical image.

[0120] For example, based on the current image 33 and the second frame image 32, the mapped image of the current image is determined.

[0121] Optionally, determining a mapped image of the current image based on the current image and the target historical image includes: determining a mapping relationship between the current image and the target historical image based on the position information of the target object that meets preset conditions in the current image and the position information of the target object that meets preset conditions in the target historical image; and mapping the current image based on the mapping relationship to obtain a mapped image of the current image.

[0122] For example, based on the position information of target object 11 in the current image 33 and the position information of target object 11 in the second frame image 32, the mapping relationship between the current image 33 and the second frame image 32 is determined. Let the position information of target object 11 in the current image 33 be denoted as (x... t ,y t The position information of target object 11 in the second frame image 32 is denoted as (x t -τ ,y t-τ ), where τ represents the time interval between the current image 33 and the second frame image 32. The mapping relationship between the current image 33 and the second frame image 32 can be denoted as: Furthermore, according to the following formula (3), it can be calculated that

[0123]

[0124] in, In other words, calculate according to formula (3) The process is actually calculation Each element in the array, for example m 11 m 12 m 13 m 21 m 22 m 23 The process.

[0125] Furthermore, the formula (3) is used to calculate... The current image 33 is mapped to obtain a mapped image 34. This ensures that the position of the target object 11 in the mapped image 34 is the same as the position of the target object 11 in the second frame image 32.

[0126] It is understood that in other embodiments, the mapping image of the current image can also be determined in other ways. For example, the current image can be input into a Generative Adversarial Network (GAN), which can be pre-trained so that the GAN can output the mapping image of the current image. In this embodiment, the mapping image of the current image can also be referred to as the enhanced image or enhanced frame of the current image.

[0127] It is understandable that, for target objects 11, 12, and 13 in the current image 33, the Ω corresponding to target object 11 is... iThe probability of target object 11 corresponding to the same identifier in its historical motion trajectory is minimized. Therefore, a relatively accurate historical motion trajectory, namely the historical motion trajectory of target object 11, is selected. Furthermore, based on the historical motion trajectory of target object 11, at least one historical image in which target object 11 appeared before the current image 33 is determined, and from this at least one historical image, the target historical image with the highest object uncertainty (e.g., image 32) is selected. Since the uncertainty of a target object is inversely proportional to the similarity between target objects with the same identifier and directly proportional to the similarity between target objects with different identifiers, it indicates that the distance between the calculated representation vector of target object 11 in the target historical image (e.g., image 32) and the representation vector of target object 11 in image 31 is not close enough, and the distance between the representation vector of target object 11 in image 32 and the representation vector of target object 12 in image 31 is not far enough. In other words, the representation vectors of target objects calculated from the target historical images cannot achieve sufficiently high similarity between target objects with the same identifier, nor sufficiently low similarity between target objects with different identifiers. The purpose of training the machine learning model in this embodiment is to ensure sufficiently high similarity between target objects with the same identifier and sufficiently low similarity between target objects with different identifiers. In other words, the role of the target historical image is exactly the opposite of the training objective. Therefore, this embodiment can use the target historical image for training, so that the trained machine learning model can more accurately encode each target object in the target historical image, making the similarity between target objects with the same identifier sufficiently high and the similarity between target objects with different identifiers sufficiently low. Specifically, when this embodiment uses the target historical image for training, the mapping relationship between the current image and the target historical image can be calculated according to the formula (3) mentioned above. Then, according to the mapping relationship, the current image is mapped to obtain the mapped image of the current image, and the position of target object 11 in the mapped image is the same as the position of target object 11 in the target historical image. Since the role of the target historical image is exactly the opposite of the training objective, the role of the mapped image is also exactly the opposite of the training objective. Therefore, the first positive sample and the first negative sample corresponding to each target object in the current image can be obtained from the mapped image. For example Figure 3As shown, for the target object marked as 1 in the current image 33, the target object marked as 1 in the mapped image 34 is the first positive sample, and the target objects marked as 2 and 3 in the mapped image 34 are the first negative samples. Similarly, for the target object marked as 2 in the current image 33, the target object marked as 2 in the mapped image 34 is the first positive sample, and the target objects marked as 1 and 3 in the mapped image 34 are the first negative samples. For the target object marked as 3 in the current image 33, the target object marked as 3 in the mapped image 34 is the first positive sample, and the target objects marked as 1 and 2 in the mapped image 34 are the first negative samples.

[0128] Furthermore, based on the representation vectors of each target object in the current image 33, and the representation vectors of the first positive sample and the first negative sample corresponding to each target object, a loss function l is constructed. The loss function L can be expressed as the following formula (4):

[0129]

[0130] Where q represents the representation vector of any target object in the current image, and k + Let represent the representation vector of the first positive sample corresponding to any target object, and Τ represent known parameters. m represents the number of the first negative samples corresponding to any target object. When i = 0, k i =k + When i ≠ 0, k i =k - k - Let q·k represent the representation vector of the first negative sample corresponding to any target object. + It can be used to represent the similarity between any target object and the first positive sample, q·k - This can be used to represent the similarity between any target object and the first negative sample. In some embodiments, the first positive sample can be denoted as a hard positive sample, and the first negative sample can be denoted as a hard negative sample.

[0131] It is understandable that the loss function L shown in formula (4) is the cumulative result after traversing each target object in the current image. However, in some other embodiments, when calculating the loss function L, it is not necessary to accumulate, but to calculate it based on a target object in the current image. Specifically, the target object can be a target object in the current image that meets the preset conditions as described above. Alternatively, the target object can also be other target objects in the current image, without specific limitations here.

[0132] In addition, such as Figure 3 The numbers 11, 12, 13, and... shown are... Figure 5The reference numerals A, B, and C shown are added to clearly describe the method of this embodiment and to clearly refer to each target object during the description of the method. These reference numerals are unknown to the electronic device, such as server 12, performing the method described in this embodiment. Server 12 can perform target recognition on each frame of the sample video to obtain the category and location of the target objects included in each frame. Then, based on the target area occupied by the target object in the image, the target area is encoded to obtain the representation vector of the target object. Then, based on the representation vector of each target object in each frame, the identifier of each target object in each frame is determined, such as 1, 2, 3, etc. This allows server 12 to regard target objects with the same identifier in different images as the same target object. However, during the training phase of the machine learning model, the accuracy of the machine learning model is not high, making the representation vectors of each target object not accurate enough. For example, the distance between the representation vectors of target objects with the same identifier may not be small enough, and the distance between the representation vectors of target objects with different identifiers may not be large enough, resulting in inaccurate identification of each target object. That is to say, during the training phase, the target objects with the same identifier in different images determined by server 12 may not actually be the same target object in reality. Therefore, the machine learning model needs to be trained using the training method described in this embodiment, so that the server 12 can identify the same target object in different images through the trained machine learning model, which is actually the same target object in reality.

[0133] It is understood that, in the above embodiments, when training with the target historical image, the mapping relationship between the current image and the target historical image can be calculated according to the formula (3) described above. Then, the current image is mapped according to the mapping relationship to obtain the mapped image of the current image. Further, training is performed based on the first positive sample and the first negative sample in the mapped image. In some other embodiments, when training with the target historical image, training can be performed not only based on the first positive sample and the first negative sample in the mapped image, but also based on the second positive sample and the second negative sample in the target historical image.

[0134] Optionally, the machine learning model to be trained may be unsupervised trained at least based on the similarity between the target object and the first positive sample, and the similarity between the target object and the first negative sample, including, for example... Figure 8 The following steps are shown:

[0135] S801. Determine the second positive sample and the second negative sample corresponding to the target object. The second positive sample is a target object in the target historical image that has the same identifier as the target object, or the second positive sample is a target object randomly selected from a first set. The first set includes target objects in at least one historical image that have the same identifier as the target object. The second negative sample is a target object in the target historical image that has a different identifier than the target object.

[0136] like Figure 3 As shown, the second frame image 32 is the target historical image corresponding to the current image 33. Therefore, in this embodiment, the second positive sample and the second negative sample corresponding to each target object in the current image 33 can also be determined from the second frame image 32. For example, for the target object marked as 1 in the current image 33, the target object marked as 1 in the second frame image 32 is the second positive sample, and the target objects marked as 2 and 3 in the second frame image 32 are the second negative samples. Similarly, the second positive sample and the second negative sample corresponding to the target object marked as 2 in the current image 33, and the second positive sample and the second negative sample corresponding to the target object marked as 3 in the current image 33 can be determined.

[0137] In other embodiments, the second positive sample corresponding to each target object in the current image 33 can also be determined by the following method. Taking the target object marked as 1 in the current image 33 as an example, at least one historical image that includes the target object marked as 1 before the current image 33 is determined. For example, if there are multiple historical images that include the target object marked as 1 before the current image 33, the target objects marked as 1 in each of these multiple historical images are used to form a first set, and a target object randomly selected from the first set is used as the second positive sample corresponding to the target object marked as 1 in the current image 33.

[0138] S802. Construct a loss function based at least on the similarity between the target object and the first positive sample, the similarity between the target object and the second positive sample, the similarity between the target object and the first negative sample, and the similarity between the target object and the second negative sample.

[0139] For example, in this embodiment, a loss function can be constructed based on the similarity between each target object in the current image 33 and its corresponding first positive sample, second positive sample, first negative sample, and second negative sample. This loss function can refer to formula (4) as described above; however, in this embodiment, k... + It can represent not only the representation vector of the first positive sample, but also the representation vector of the second positive sample. -It can represent not only the representation vector of the first negative sample, but also the representation vector of the second negative sample. Accordingly, m represents the number of the first and second negative samples corresponding to any target object.

[0140] S803. Perform unsupervised training on the machine learning model to be trained according to the loss function.

[0141] For example, referring to formula (4) as described above, unsupervised training is performed on the machine learning model to be trained. Specifically, the loss value of the loss function L can be calculated according to formula (4) as described above, and the parameters of the machine learning model to be trained can be adjusted according to the loss value. For example, the parameters of the encoder in the machine learning model to be trained can be adjusted so that the adjusted encoder can make the distance between the representation vectors of target objects with the same label closer, while making the distance between the representation vectors of target objects with different labels farther.

[0142] Optionally, after determining the second positive sample and the second negative sample corresponding to the target object, the method further includes: determining the third negative sample corresponding to the target object, wherein the third negative sample is at least one target object in a second set, the second set including at least one of the following: target objects with different identifiers corresponding to the target object in historical images other than the target historical image in at least one historical image, target objects with different identifiers corresponding to the target object in the mapping images corresponding to the at least one historical image, target objects with different identifiers corresponding to the target object in the current image, and target objects appearing in other videos different from the sample video.

[0143] like Figure 3 As shown, the second frame image 32 is the target historical image corresponding to the current image 33. The above embodiment can determine the second positive sample and the second negative sample corresponding to each target object in the current image 33 from the second frame image 32. However, for any target object in the current image 33, other historical images besides the second frame image 32 may also contain target objects with different identifiers. Here, target objects with different identifiers corresponding to any target object appearing in other historical images besides the second frame image 32 can be placed into a second set.

[0144] In addition, the above embodiments describe a mapped image of the current image, for example, Figure 3The mapped image of the current image 33 is shown as image 34, and the method for generating the mapped image is introduced. It is understood that a similar generation method can be used to generate mapped images for each historical image preceding the current image 33; that is, each historical image preceding the current image 33 can also correspond to a mapped image. For any target object in the current image 33, it is assumed that at least one historical image preceding the current image 33 contains a target object with a different identifier than that target object in its corresponding mapped image. Here, the target objects with different identifiers appearing in the mapped images corresponding to at least one historical image can be placed into a second set.

[0145] Furthermore, for any target object in the current image 33, other target objects in the current image 33 correspond to different identifiers than any target object. Therefore, the target objects in the current image 33 that correspond to different identifiers than any target object can also be placed into the second set.

[0146] In some other embodiments, server 22 may also process other videos besides the sample video to identify target objects appearing in other videos and place the target objects appearing in other videos into a second set.

[0147] It is understood that the target objects in the second set are not limited to the possible sources mentioned above, and other sources may also exist in other embodiments. Furthermore, among the sources described above, the target objects in the second set originate from at least one source.

[0148] Furthermore, at least one target object can be obtained from the second set, and the at least one target object can be used as the third negative sample corresponding to any target object.

[0149] Constructing a loss function based at least on the similarity between the target object and the first positive sample, the target object and the second positive sample, the target object and the first negative sample, and the target object and the second negative sample includes: constructing a loss function based on the similarity between the target object and the first positive sample, the target object and the second positive sample, the target object and the first negative sample, the target object and the second negative sample, and the target object and the third negative sample.

[0150] For example, the loss function can refer to formula (4) as described above; however, in this embodiment, k + It can represent not only the representation vector of the first positive sample, but also the representation vector of the second positive sample. -It can represent not only the representation vector of the first negative sample, but also the representation vector of the second negative sample and the representation vector of the third negative sample. Accordingly, m represents the number of the first, second, and third negative samples corresponding to any target object.

[0151] In addition, in other embodiments, a third positive sample corresponding to any target object can be determined. The third positive sample can be at least one target object in a third set, which includes at least one of the following: a target object with the same identifier as the target object in the historical images other than the target historical image in the at least one historical image, and a target object with the same identifier as the target object in the mapping images corresponding to the at least one historical image.

[0152] For example, such as Figure 3 As shown, the second frame image 32 is the target historical image corresponding to the current image 33. The above embodiment can determine the second positive sample and the second negative sample corresponding to each target object in the current image 33 from the second frame image 32. However, for any target object in the current image 33, other historical images besides the second frame image 32 may also contain target objects with the same identifier as that target object. Here, target objects with the same identifier as that target object appearing in other historical images besides the second frame image 32 can be placed into a third set.

[0153] Additionally, each historical image preceding the current image 33 can also correspond to a mapping image. For any target object in the current image 33, assume that at least one historical image preceding the current image 33 contains a target object with the same identifier as that target object in its corresponding mapping image. Here, the target objects with the same identifier as that target object appearing in the mapping images corresponding to the at least one historical image can be placed into a third set.

[0154] Therefore, in other embodiments, a loss function can also be constructed based on the similarity between the target object and the first positive sample, the similarity between the target object and the second positive sample, the similarity between the target object and the third positive sample, the similarity between the target object and the first negative sample, the similarity between the target object and the second negative sample, and the similarity between the target object and the third negative sample.

[0155] For example, the loss function can refer to formula (4) as described above; however, in this embodiment, k + It can represent not only the representation vector of the first positive sample, but also the representation vector of the second positive sample and the representation vector of the third positive sample. -It can represent not only the representation vector of the first negative sample, but also the representation vector of the second negative sample and the representation vector of the third negative sample. Accordingly, m represents the number of the first, second, and third negative samples corresponding to any target object.

[0156] The machine learning model to be trained, as described above, can be an uncertainty-aware unsupervised MOT framework. This framework consists of two modules: an uncertainty-aware tracklet labeling (UTL) module and a tracklet-guided augmentation (TGA) module, aiming to maintain consistency within tracklets while improving the distinguishability and discriminability between tracklets. The following section will combine... Figure 9 The structure of the framework and its training process are described in detail.

[0157] like Figure 9 As shown, box 91 corresponds to the UTL module, box 92 corresponds to the TGA module, and box 93 represents the contrastive learning process. Specifically, T0 represents the time corresponding to the first frame of the sample video, T1 represents the time corresponding to the second frame of the sample video, and T... τThis represents the current time frame in the sample video. Assume server 22 can perform target recognition on each frame of the sample video, thereby obtaining the category and location of each target object in each frame. Then, an encoder encodes the target region occupied by each target object in each frame, obtaining a representation vector for each target object in each frame. For example, the circles output by the encoder represent the representation vectors of the target objects. Then, based on the representation vectors of each target object in each frame, the identifier of each target object in each frame is determined. The identifiers of each target object in the first frame are initialized. The first module determines the initial identifiers of each target object in the second frame based on the representation vectors of each target object in the second frame and the identifiers of each target object in the first frame, such as the identifiers in box 94. Specifically, the first module can calculate the similarity between each target object in the second frame and each target object in the first frame based on the representation vectors of each target object in the second frame and the representation vectors of each target object in the first frame. The calculation result can be a similarity matrix, for example, a 3x3 matrix. Then, based on this similarity matrix, the initial identifiers of each target object in the second frame are determined. Here, the target objects in the second frame image are associated with the target objects in the first frame image based on their appearance similarity. However, the appearance similarity between target objects is unreliable, especially for appearance features (such as the representation vectors of target objects) learned in unsupervised scenarios. Therefore, this embodiment adds processing by a second module and a third module. Specifically, the second module can calculate the uncertainty of the initial identifier corresponding to each target object in the second frame image according to the formula (1) described above, and determine the initial identifiers with uncertainties greater than or equal to a preset threshold, such as the identifier in box 95. Then, the third module can correct the identifiers with uncertainties greater than or equal to the preset threshold. In this way, through the first module, the second module, and the third module, the identifiers of each target object in the first frame image can be propagated to the second frame image. Similarly, propagation can continue until it reaches the last frame image in the sample video. Since the uncertainty is calculated by the second module and the identification is corrected by the third module during each propagation, accurate pseudo-label tracks can be generated after the identification is propagated frame by frame. These pseudo-label tracks can be the identifiers of each target object in each frame of the image, as shown in box 96. The same identifier can constitute a single pseudo-label track. The second and third modules improve consistency within the tracklet, ensuring that the same target object corresponds to the same identifier in different images or in historical motion trajectories, thus making the representation vectors of target objects with the same identifier as consistent as possible.

[0158] After identifying the identifiers of each target object in each frame of the sample video, the current image is selected from the sample video; for example, the current image is the last frame. At this point, the target objects in the current image can be distinguished from the target objects in previous historical images. For example, the representation vectors of each target object in the current image are represented by dotted circles, while the representation vectors of each target object in previous historical images are represented by solid circles. Figure 9 The anchor selection process described employs a hierarchical uncertainty sampling mechanism. Specifically, in this process, the tracklet uncertainty of each target object in the current image is calculated according to formula (2) as described above. Tracklet uncertainty can be used to evaluate the accuracy or reliability of the pseudo-label trajectory. For example, the target object with the smallest tracklet uncertainty is first determined. Assuming that the tracklet uncertainty of the target object labeled 2 in the current image is the smallest, the target object labeled 2 in the current image can be used as the target object that meets the preset conditions as described above. This ensures that the pseudo-label trajectory labeled 2 is an accurate trajectory, that is, it ensures that the probability of the target object labeled 2 corresponding to the same label in different images is the largest. Then, along the pseudo-label trajectory labeled 2, a target historical image with the largest object uncertainty is selected from the historical images in which the target object labeled 2 has appeared. For example, if the object uncertainty corresponding to the target object labeled 2 is the largest in the second frame image, then the second frame image can be used as the target historical image.

[0159] Furthermore, according to formula (3), the mapping relationship between the current image and the target historical image is calculated, and the current image is mapped according to this mapping relationship to obtain the mapped image of the current image. Specifically, Figure 9 The input to the TGA shown can be the position of the target object identified as 2 in the current image, and the position of the target object identified as 2 in the target's historical image. The TGA calculates the mapping relationship based on these two positions and formula (3), and then maps the current image according to this mapping relationship, thereby outputting a mapped image of the current image, for example... Figure 9Image 97 is shown. Then, an encoder is used to encode each target object in image 97, obtaining the representation vector of each target object in image 97. Furthermore, to distinguish between target objects in the mapped image, target objects in the current image, and target objects in historical images, the representation vectors of target objects in the mapped image are represented by dashed circles, the representation vectors of target objects in the current image are represented by dotted circles, and the representation vectors of target objects in historical images are represented by solid circles.

[0160] Then, comparative learning is performed using formula (4). Here, q in formula (4) represents the representation vector of any target object in the current image, for example, the representation vector of the target object identified as 2 in the current image. + This includes the representation vector of the target object identified as 2 in the mapped image, and the representation vector of the target object identified as 2 in the target historical image. - This includes the representation vector of the target object whose identifier is not 2 in the mapped image, and the representation vector of the target object whose identifier is not 2 in the target historical image. Optionally, in some other embodiments, k + It may further include the representation vector of the target object identified as 2 in other historical images besides the target historical image, and / or further include the representation vector of the target object identified as 2 in the mapping images corresponding to each historical image preceding the current image. Optionally, in some other embodiments, k - It may further include at least one of the following: representation vectors of target objects with a label not equal to 2 in historical images other than the target historical image; representation vectors of target objects with a label not equal to 2 in the mapping images corresponding to each historical image preceding the current image; representation vectors of target objects with a label not equal to 2 in the current image; and representation vectors of target objects appearing in other videos different from the sample video. For example... Figure 9 The 98 shown represents other videos, and the set 99 includes the representation vectors of multiple target objects that appear in other videos.

[0161] In this embodiment, the representation vector of each target object in the mapped image can be denoted as the augmented key, the representation vector of each target object in the historical image can be denoted as the historical key, the representation vector of each target object appearing in other videos can be denoted as the auxiliary key, and other videos can also be denoted as auxiliary videos.

[0162] In this embodiment, the contrastive learning shown in formula (4) can be used to... Figure 9 The encoder is trained using a solid wireframe, for example, based on q·k. +It can guide the encoder to improve the similarity between target objects with the same identifier, based on q·k - This can guide the encoder to reduce the similarity between target objects with different identifiers. Therefore, according to Figure 9 The TGA module and contrastive learning shown not only improve consistency within tracklets but also enhance the discriminability between tracklets. Discriminability between tracklets refers to making the representation vectors of target objects with different identifiers as different as possible, thus distinguishing them from each other. In other words, improving consistency within tracklets improves the consistency within sets 910, 911, and 912. Improving discriminability between tracklets improves the discriminability between sets 910, 911, and 912. Furthermore, this embodiment can also mine hard positive and hard negative samples based on a hierarchical uncertainty adoption mechanism, thereby realizing an effective hard sample mining method.

[0163] It is understandable that after the encoder described above is trained, it can be applied to different scenarios. In one possible scenario, it can be applied to the field of autonomous driving. For example, an autonomous vehicle may be equipped with a camera that captures images of the vehicle's surroundings, creating a target video. This target video may contain obstacles, which can be considered target objects. Since the autonomous vehicle is constantly changing relative to the obstacles, by encoding each frame of the target video using the encoder, the representation vector of each obstacle in each frame can be obtained. Furthermore, based on the representation vector of each obstacle in each frame, the identifier of each obstacle in each frame can be determined. This allows for accurate analysis and judgment of obstacles, thus helping the autonomous vehicle to avoid obstacles more precisely. In another possible scenario, it can also be used for intelligent robots. For example, in a goods sorting scenario, by identifying the goods in the video captured by the intelligent robot, it helps the robot accurately lock onto the goods to be sorted during movement. It is understood that the method described in this embodiment can be used in other scenarios requiring target object recognition and identification, not just the scenarios mentioned above. The following section details the usage or inference phases of a trained encoder or machine learning model.

[0164] Figure 10 This is a flowchart illustrating a video processing method according to another embodiment of the present disclosure. Specifically, this method can be executed by a terminal or by a cloud server. This embodiment uses terminal execution as an example for illustrative explanation. In this embodiment, the specific steps of the method are as follows:

[0165] S1001. Obtain the target video, which includes multiple frames of images.

[0166] For example, server 22 can deploy a trained encoder or a trained machine learning model on a terminal. The terminal can capture the target video in real time, or it can receive the target video sent by other servers, which includes multiple frames of images. Alternatively, the target video can also be a video stream.

[0167] S1002. For each frame of the multi-frame images, at least one target region in the image is encoded by a machine learning model to obtain a representation vector corresponding to at least one target object in the image. The target region includes one or more target objects. The machine learning model is trained by the method described above.

[0168] For example, for each frame of the target video, the terminal can encode at least one target region in the image using a trained encoder or a trained machine learning model, obtaining a representation vector corresponding to at least one target object in the image. Specifically, a target region includes one or more target objects. Because the trained encoder or trained machine learning model has high accuracy, the representation vectors corresponding to each target object in each frame are relatively accurate, meaning that the representation vectors of the same target object have high consistency in different images, and the representation vectors of different target objects have high discriminative power.

[0169] S1003. Determine the identifiers corresponding to at least one target object in the image based on the representation vectors corresponding to at least one target object in the image.

[0170] For example, based on the representation vectors corresponding to at least one target object in each frame of the target video, the identifiers corresponding to at least one target object in the image are determined. Specifically, the process can employ the first module described above to determine the identifiers corresponding to at least one target object in the image. For instance, a historical image frame adjacent to the current image is selected, and based on the representation vectors corresponding to at least one target object in the current image and the representation vectors corresponding to at least one target object in the historical image, the similarity (e.g., appearance similarity) between each target object in the current image and each target object in the historical image is calculated, resulting in a similarity matrix. Then, based on this similarity matrix, the initial identifier for each target object in the current image is determined. For example, the initial identifier for any target object in the current image is determined based on the appearance similarity between that target object and each target object in the historical image; the identifier of the target object in the historical image with the highest appearance similarity to that target object is used as the identifier for that target object, thus propagating the identifiers of the target objects in the historical images to the target objects in the current image. However, appearance similarity between target objects is unreliable; therefore, it can be corrected through the following two steps to improve the accuracy of the identifiers.

[0171] S1004. Based on the identifiers corresponding to at least one target object in the image and the identifiers corresponding to at least one target object in previous historical images, calculate the uncertainty of the identifier corresponding to each target object in the image. The uncertainty is related to the similarity between target objects with the same identifier and the similarity between target objects with different identifiers.

[0172] For example, the second module can calculate the uncertainty of the initial label corresponding to each target object in the image according to formula (1) as described above. This uncertainty is related to the similarity between target objects with the same label and the similarity between target objects with different labels. Furthermore, the second module can also determine the initial labels in the image whose uncertainty is greater than or equal to a preset threshold, for example... Figure 9 The label in box 95 shown.

[0173] S1005. Correct the markings in the image whose uncertainty is greater than or equal to a preset threshold.

[0174] For example, the third module can correct identifiers with uncertainties greater than or equal to a preset threshold. Specifically, the correction method may be to redetermine the identifier of the target object with an uncertainty greater than or equal to the preset threshold, or to correct the identifier of the target object based on the appearance similarity between the target object in multiple historical images preceding the image and the target object in the current image, as well as the historical motion trajectory of the target object with an uncertainty greater than or equal to the preset threshold in the current image. It is understood that the correction methods are not limited to the correction methods described in this embodiment; in other embodiments, other correction methods may also be used.

[0175] This embodiment encodes at least one target region in each frame of the target video using a high-precision encoder or machine learning model, obtaining a representation vector corresponding to at least one target object in each frame. This ensures that the representation vectors for each target object in each frame are highly accurate, meaning that the representation vectors for the same target object have high consistency across different images, and the representation vectors for different target objects have high distinguishability. Furthermore, based on the accurate representation vectors of each target object in each frame, the identifier of each target object in each frame can be accurately determined. Additionally, by calculating the uncertainty of the identifier corresponding to each target object in the image and correcting identifiers with uncertainty greater than or equal to a preset threshold, the accuracy of the identifiers for each target object in each frame can be further improved. This ensures that the same target object corresponds to the same identifier in different images and accurately distinguishes different target objects. This effectively avoids identification errors or switching between adjacent identifiers caused by target object occlusion or camera shake.

[0176] Figure 11 This is a flowchart illustrating a video processing method according to another embodiment of the present disclosure. Specifically, this method can be executed by a terminal or by a cloud server. This embodiment uses execution by a cloud server as an example for illustrative purposes. In this embodiment, the specific steps of the method are as follows:

[0177] S1101. Receive vehicle video and obtain multiple frames of images from the vehicle video.

[0178] For example, a cloud server may be deployed with a trained encoder or a trained machine learning model as described above. The cloud server receives vehicle video in real time, which could be video of other vehicles around the vehicle captured by a camera mounted on the vehicle. Alternatively, the vehicle video could be video of the vehicle captured by a camera fixed to traffic infrastructure such as a traffic light bracket.

[0179] S1102. For each frame of the multi-frame images, at least one target region in the image is encoded by a machine learning model to obtain a representation vector corresponding to at least one vehicle in the image. The target region includes one or more vehicles. The machine learning model is trained by the training method described above.

[0180] For example, for each frame of the vehicle video, the cloud server can use a trained encoder or a trained machine learning model to encode at least one target region in the image, obtaining the representation vector corresponding to at least one vehicle in the image. Specifically, a target region includes one or more vehicles. Because the trained encoder or trained machine learning model has high accuracy, the representation vectors corresponding to each vehicle in each frame are relatively accurate, meaning that the representation vectors of the same vehicle in different images have high consistency, and the representation vectors of different vehicles have high discriminative power.

[0181] S1103. Based on the representation vectors corresponding to at least one vehicle in the image, determine the identifiers corresponding to at least one vehicle in the image.

[0182] For example, based on the representation vectors corresponding to at least one vehicle in each frame of a vehicle video image, the identifiers corresponding to at least one vehicle in the image are determined. Specifically, the process can employ the first module described above to determine the identifiers corresponding to at least one vehicle in the image. For instance, a historical image frame adjacent to the current image is selected, and based on the representation vectors corresponding to at least one vehicle in the current image and the representation vectors corresponding to at least one vehicle in the historical image, the similarity (e.g., appearance similarity) between each vehicle in the current image and each vehicle in the historical image is calculated, resulting in a similarity matrix. Then, based on this similarity matrix, the initial identifier for each vehicle in the current image is determined. For example, the initial identifier for any vehicle in the current image is determined based on the appearance similarity between that vehicle and each vehicle in the historical image; the identifier of the vehicle with the highest appearance similarity to that vehicle in the historical image is used as the identifier for that vehicle, thus propagating the identifiers of vehicles in the historical image to vehicles in the current image. However, the appearance similarity between vehicles is unreliable; therefore, it can be corrected through the following two steps to improve the accuracy of the identifiers.

[0183] S1104. Based on the identifiers corresponding to at least one vehicle in the image and the identifiers corresponding to at least one vehicle in previous historical images, calculate the uncertainty of the identifier corresponding to each vehicle in the image. The uncertainty is related to the similarity between vehicles with the same identifier and the similarity between vehicles with different identifiers.

[0184] For example, the second module can calculate the uncertainty of the initial identifier corresponding to each vehicle in the image according to formula (1) as described above. This uncertainty is related to the similarity between vehicles with the same identifier in the image and the historical image, and also to the similarity between vehicles with different identifiers in the image and the historical image. Furthermore, the second module can also determine the initial identifiers in the image whose uncertainty is greater than or equal to a preset threshold, for example... Figure 9 The label in box 95 shown.

[0185] S1105. Correct the markings in the image whose uncertainty is greater than or equal to a preset threshold.

[0186] For example, the third module can correct identifiers with uncertainty greater than or equal to a preset threshold. Specifically, the correction method may be to redetermine the identifiers of vehicles with uncertainty greater than or equal to the preset threshold, or to correct the vehicle identifiers based on the appearance similarity of vehicles in multiple historical images preceding the image and the vehicles in the current image, as well as the historical motion trajectories of vehicles in the image with uncertainty greater than or equal to the preset threshold. It is understood that the correction methods are not limited to the correction methods described in this embodiment; in other embodiments, other correction methods may also be used.

[0187] This embodiment encodes at least one target region in each frame of a vehicle video image using a high-precision encoder or machine learning model, obtaining a representation vector corresponding to at least one vehicle in each frame. This ensures the accuracy of the representation vectors for each vehicle in each frame, resulting in high consistency between the representation vectors of the same vehicle across different images and high distinguishability between the representation vectors of different vehicles. Furthermore, based on the precise representation vectors of each vehicle in each frame, the identifier of each vehicle in each frame can be accurately determined. Additionally, by calculating the uncertainty of the identifier corresponding to each vehicle in the image and correcting identifiers with uncertainty greater than or equal to a preset threshold, the accuracy of the vehicle identifiers in each frame can be further improved. This ensures that the same vehicle corresponds to the same identifier in different images and accurately distinguishes different vehicles. This effectively avoids identification errors or switching between adjacent identifiers caused by vehicle occlusion, camera shake, or other issues.

[0188] Figure 12 This is a flowchart illustrating a video processing method according to another embodiment of the present disclosure. Specifically, this method can be executed by a terminal or by a cloud server. This embodiment uses execution by a cloud server as an example for illustrative purposes. In this embodiment, the specific steps of the method are as follows:

[0189] S1201. Receive a video of a person and obtain multiple frames of images from the video of the person.

[0190] For example, a cloud server may be equipped with a trained encoder or a trained machine learning model, as described above. The cloud server receives real-time videos of people, which may be videos containing people captured by a camera.

[0191] S1202. For each frame of the multi-frame images, at least one target region in the image is encoded by a machine learning model to obtain a representation vector corresponding to at least one person in the image. The target region includes one or more people. The machine learning model is trained by the training method described above.

[0192] For example, for each frame of a video of a person, the cloud server can use a trained encoder or a trained machine learning model to encode at least one target region in the image, obtaining a representation vector corresponding to at least one person in the image. Specifically, a target region includes one or more people. Because the trained encoder or trained machine learning model has high accuracy, the representation vectors corresponding to each person in each frame are relatively accurate, meaning that the representation vectors of the same person in different images have high consistency, and the representation vectors of different people have high discriminative power.

[0193] S1203. Determine the identifiers corresponding to at least one person in the image based on the representation vectors corresponding to at least one person in the image.

[0194] For example, based on the representation vectors corresponding to at least one person in each frame of a video image, the identifiers corresponding to at least one person in that image are determined. Specifically, the process can employ the first module described above to determine the identifiers corresponding to at least one person in the image. For instance, a historical image adjacent to the current image is selected, and based on the representation vectors corresponding to at least one person in the current image and the representation vectors corresponding to at least one person in the historical image, the similarity (e.g., appearance similarity) between each person in the current image and each person in the historical image is calculated, resulting in a similarity matrix. Then, based on this similarity matrix, the initial identifier for each person in the current image is determined. For example, the initial identifier for any person in the current image is determined based on the appearance similarity between that person and each person in the historical image; the identifier of the person in the historical image with the highest appearance similarity to that person is used as the identifier for that person, thus propagating the identifiers of people in the historical image to people in the current image. However, appearance similarity between people is unreliable; therefore, it can be corrected through the following two steps to improve the accuracy of the identifiers.

[0195] S1204. Based on the identifiers corresponding to at least one person in the image and the identifiers corresponding to at least one person in previous historical images, calculate the uncertainty of the identifier corresponding to each person in the image. The uncertainty is related to the similarity between people with the same identifier and the similarity between people with different identifiers.

[0196] For example, the second module can calculate the uncertainty of the initial identifier corresponding to each person in the image according to formula (1) as described above. This uncertainty is related to the similarity between people with the same identifier in the image and the historical image, and also to the similarity between people with different identifiers in the image and the historical image. Furthermore, the second module can also determine the initial identifiers in the image whose uncertainty is greater than or equal to a preset threshold, for example... Figure 9 The label in box 95 shown.

[0197] S1205. Correct the markings in the image whose uncertainty is greater than or equal to a preset threshold.

[0198] For example, the third module can correct identifiers with uncertainty greater than or equal to a preset threshold. Specifically, the correction method may be to redetermine the identifiers of people with uncertainty greater than or equal to the preset threshold, or to correct the identifier of a person based on the appearance similarity between people in multiple historical images preceding the image and the people in the current image, as well as the historical movement trajectory of people in the image with uncertainty greater than or equal to the preset threshold. It is understood that the correction methods are not limited to the correction methods described in this embodiment; in other embodiments, other correction methods may also be used.

[0199] This embodiment encodes at least one target region in each frame of a video image using a high-precision encoder or machine learning model, obtaining a representation vector corresponding to at least one person in each frame. This ensures the accuracy of the representation vectors for each person in each frame, meaning the representation vectors for the same person in different images have high consistency, and the representation vectors for different people have high distinguishability. Furthermore, based on the accurate representation vectors of each person in each frame, the identifier of each person in each frame can be precisely determined. Additionally, by calculating the uncertainty of the identifier for each person in the image and correcting identifiers with uncertainty greater than or equal to a preset threshold, the accuracy of the identifiers for each person in each frame can be further improved. This ensures that the same person corresponds to the same identifier in different images and accurately distinguishes different people. This effectively avoids identification errors or switching between adjacent identifiers caused by occlusion of the person or camera shake.

[0200] Figure 13This is a schematic diagram of a machine learning model training apparatus provided in an embodiment of this disclosure. The machine learning model training apparatus provided in this embodiment can execute the processing flow provided in the machine learning model training method embodiment, such as… Figure 13 As shown, the machine learning model training device 110 includes:

[0201] The encoding module 111 is used to encode at least one target region in each frame of the sample video using a machine learning model to be trained, so as to obtain a representation vector corresponding to at least one target object in the image, wherein the target region includes one or more target objects.

[0202] The first determining module 112 is used to determine the identifiers corresponding to at least one target object in the image based on the representation vectors corresponding to at least one target object in the image.

[0203] The second determining module 113 is used to select the current image from the sample video, determine the historical motion trajectory of each target object in the current image based on the identifiers corresponding to at least one target object in the current image and the identifiers corresponding to target objects in at least one historical image before the current image, and determine the mapping image of the current image based on the historical motion trajectory of each target object in the current image. The mapping image of the current image includes a first positive sample and a first negative sample corresponding to each target object in the current image.

[0204] The training module 114 is used to traverse each target object in the current image and perform unsupervised training on the machine learning model to be trained based at least on the similarity between the target object and the first positive sample and the similarity between the target object and the first negative sample.

[0205] Optionally, when the first determining module 112 determines the identifiers corresponding to at least one target object in the image based on the representation vectors corresponding to at least one target object in the image, it is specifically used for:

[0206] If the image is the first image in the sample video to contain a target object, then each target object in the image is assigned a different identifier based on the representation vector corresponding to at least one target object in the image.

[0207] Optionally, when the first determining module 112 determines the identifiers corresponding to at least one target object in the image based on the representation vectors corresponding to at least one target object in the image, it is specifically used for:

[0208] If the image is not the first image in the sample video to contain the target object, then the identifier corresponding to at least one target object in the image is determined based on the representation vector corresponding to at least one target object in the image and the identifier corresponding to at least one target object in the historical images preceding the image.

[0209] Based on the identifiers corresponding to at least one target object in the image and the identifiers corresponding to at least one target object in previous historical images, the uncertainty of the identifier corresponding to each target object in the image is calculated. The uncertainty is related to the similarity between target objects with the same identifier and the similarity between target objects with different identifiers.

[0210] Correct the identifiers in the image whose uncertainty is greater than or equal to a preset threshold.

[0211] Optionally, when the first determining module 112 determines the identifiers corresponding to at least one target object in the image based on the representation vectors corresponding to at least one target object in the image and the identifiers corresponding to at least one target object in previous historical images, it is specifically used for:

[0212] Based on the representation vectors corresponding to at least one target object in the image and the representation vectors corresponding to at least one target object in the previous historical images, the similarity between each target object in the image and each target object in the historical images is determined.

[0213] Based on the similarity between each target object in the image and each target object in the historical image, and the identifiers corresponding to at least one target object in the historical image, the identifiers corresponding to at least one target object in the image are determined.

[0214] Optionally, the first determining module 112 calculates the uncertainty of each target object's corresponding identifier in the image based on the identifiers corresponding to at least one target object in the image and the identifiers corresponding to at least one target object in previous historical images, including:

[0215] Based on the identifier corresponding to any target object in the image, determine a target object in a previous historical image that has the same identifier as the target object, and multiple target objects in the historical image that have different identifiers than the target object;

[0216] The uncertainty of the identifier corresponding to any target object is calculated based on the similarity between the target object and any target object, and the maximum value among the similarity between the multiple target objects and any target object.

[0217] Optionally, when the second determining module 113 determines the mapped image of the current image based on the historical motion trajectory of each target object in the current image, it is specifically used for:

[0218] Based on the historical motion trajectory of each target object in the current image, target objects that meet preset conditions are determined from the current image, wherein the target objects that meet the preset conditions have the highest probability of corresponding to the same identifier in their historical motion trajectory;

[0219] Based on the uncertainty of the corresponding identifier of the target object that meets the preset conditions in the at least one historical image, a target historical image is determined from the at least one historical image, wherein the uncertainty of the corresponding identifier of the target object that meets the preset conditions in the target historical image is the largest.

[0220] Based on the current image and the target historical image, determine the mapping image of the current image.

[0221] Optionally, when the second determining module 113 determines the mapped image of the current image based on the current image and the target historical image, it is specifically used for:

[0222] Based on the location information of the target object that meets the preset conditions in the current image and the location information of the target object that meets the preset conditions in the target historical image, the mapping relationship between the current image and the target historical image is determined;

[0223] Based on the mapping relationship, the current image is mapped to obtain the mapped image of the current image.

[0224] Optionally, when the training module 114 performs unsupervised training on the machine learning model to be trained based at least on the similarity between the target object and the first positive sample, and the similarity between the target object and the first negative sample, it is specifically used for:

[0225] Determine a second positive sample and a second negative sample corresponding to the target object. The second positive sample is a target object in the target historical image that has the same identifier as the target object, or the second positive sample is a target object randomly selected from a first set, the first set including at least one target object in the historical image that has the same identifier as the target object. The second negative sample is a target object in the target historical image that has a different identifier than the target object.

[0226] A loss function is constructed based at least on the similarity between the target object and the first positive sample, the similarity between the target object and the second positive sample, the similarity between the target object and the first negative sample, and the similarity between the target object and the second negative sample;

[0227] The machine learning model to be trained is subjected to unsupervised training based on the loss function.

[0228] Optionally, after determining the second positive sample and the second negative sample corresponding to the target object, the training module 114 is further configured to determine the third negative sample corresponding to the target object. The third negative sample is at least one target object in a second set, which includes at least one of the following: target objects with different identifiers corresponding to the target object in historical images other than the target historical image in the at least one historical image; target objects with different identifiers corresponding to the target object in the mapping images corresponding to the at least one historical image; target objects with different identifiers corresponding to the target object in the current image; and target objects appearing in other videos different from the sample video. When constructing the loss function based at least on the similarity between the target object and the first positive sample, the similarity between the target object and the second positive sample, the similarity between the target object and the first negative sample, and the similarity between the target object and the second negative sample, the training module 114 is specifically configured to:

[0229] A loss function is constructed based on the similarity between the target object and the first positive sample, the target object and the second positive sample, the target object and the first negative sample, the target object and the second negative sample, and the target object and the third negative sample.

[0230] Figure 13 The machine learning model training apparatus of the illustrated embodiment can be used to execute the technical solutions of the above method embodiments. Its implementation principle and technical effect are similar, and will not be repeated here.

[0231] Figure 14 This is a schematic diagram of the structure of a video processing apparatus provided in an embodiment of this disclosure. The video processing apparatus provided in this embodiment of the disclosure can execute the processing flow provided in the video processing method embodiment, such as... Figure 14 As shown, the video processing apparatus 120 includes:

[0232] Acquisition module 121 is used to acquire a target video, the target video including multiple frames of images;

[0233] Encoding module 122 is used to encode at least one target region in each frame of the multi-frame images using a machine learning model to obtain a representation vector corresponding to at least one target object in the image. The target region includes one or more target objects. The machine learning model is trained by the method described above.

[0234] The determining module 123 is used to determine the identifiers corresponding to at least one target object in the image based on the representation vectors corresponding to at least one target object in the image.

[0235] The calculation module 124 is used to calculate the uncertainty of the identifier corresponding to each target object in the image based on the identifier corresponding to at least one target object in the image and the identifier corresponding to at least one target object in the previous historical images. The uncertainty is related to the similarity between target objects with the same identifier and the similarity between target objects with different identifiers.

[0236] The correction module 125 is used to correct the markers in the image whose uncertainty is greater than or equal to a preset threshold.

[0237] Figure 14 The video processing apparatus of the illustrated embodiment can be used to execute the technical solutions of the above method embodiments, and its implementation principle and technical effect are similar, so they will not be described again here.

[0238] The above describes the internal functions and structure of a machine learning model training device or a video processing device, which can be implemented as an electronic device. Figure 15 A schematic diagram illustrating the structure of an electronic device embodiment provided in this disclosure. (See attached diagram.) Figure 15 As shown, the electronic device includes a memory 131 and a processor 132.

[0239] Memory 131 is used to store programs. In addition to the programs described above, memory 131 can also be configured to store various other data to support operation on the electronic device. Examples of this data include instructions for any application or method used to operate on the electronic device, contact data, phone book data, messages, pictures, videos, etc.

[0240] The memory 131 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0241] The processor 132 is coupled to the memory 131 and executes the program stored in the memory 131 for:

[0242] For each frame of the sample video, at least one target region in the image is encoded by the machine learning model to be trained, so as to obtain the representation vector corresponding to at least one target object in the image. The target region includes one or more target objects.

[0243] Based on the representation vectors corresponding to at least one target object in the image, determine the identifiers corresponding to at least one target object in the image;

[0244] The current image is selected from the sample video. Based on the identifiers corresponding to at least one target object in the current image and the identifiers corresponding to target objects in at least one historical image before the current image, the historical motion trajectory of each target object in the current image is determined. Based on the historical motion trajectory of each target object in the current image, the mapping image of the current image is determined. The mapping image of the current image includes a first positive sample and a first negative sample corresponding to each target object in the current image.

[0245] The machine learning model to be trained is unsupervised by traversing each target object in the current image and at least based on the similarity between the target object and the first positive sample, and the similarity between the target object and the first negative sample.

[0246] Alternatively, processor 132 is used for:

[0247] Acquire a target video, which includes multiple frames of images;

[0248] For each frame of the multi-frame images, at least one target region in the image is encoded by a machine learning model to obtain a representation vector corresponding to at least one target object in the image. The target region includes one or more target objects. The machine learning model is trained by the method described above.

[0249] Based on the representation vectors corresponding to at least one target object in the image, the identifiers corresponding to at least one target object in the image are determined.

[0250] Furthermore, such as Figure 15 As shown, the electronic device may also include other components such as a communication component 133, a power supply component 134, an audio component 135, and a display 136. Figure 15 The diagram only shows some components and does not mean that the electronic device includes only these components. Figure 15 The components shown.

[0251] Communication component 133 is configured to facilitate wired or wireless communication between electronic devices and other devices. The electronic devices can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 133 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 133 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0252] Power supply component 134 provides power to various components of an electronic device. Power supply component 134 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device.

[0253] Audio component 135 is configured to output and / or input audio signals. For example, audio component 135 includes a microphone (MIC) configured to receive external audio signals when the electronic device is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 131 or transmitted via communication component 133. In some embodiments, audio component 135 also includes a speaker for outputting audio signals.

[0254] Display 136 includes a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touchscreen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation.

[0255] In addition, this disclosure also provides a computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the methods described in the above embodiments.

[0256] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0257] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for training a machine learning model, wherein, The method includes: For each frame of the sample video, at least one target region in the image is encoded by the machine learning model to be trained, so as to obtain the representation vector corresponding to at least one target object in the image. The target region includes one or more target objects. Based on the representation vectors corresponding to at least one target object in the image, determine the identifiers corresponding to at least one target object in the image; The current image is selected from the sample video. Based on the identifiers corresponding to at least one target object in the current image and the identifiers corresponding to target objects in at least one previous historical image, the historical motion trajectory of each target object in the current image is determined. Based on the historical motion trajectories of each target object in the current image, a mapping image of the current image is determined. The mapping image of the current image includes a first positive sample and a first negative sample corresponding to the target object with the highest identifier probability in the current image. The target object with the highest identifier probability is the target object that meets the preset conditions determined from the current image based on the historical motion trajectory of each target object in the current image. The target object that meets the preset conditions has the highest probability of corresponding to the same identifier in its historical motion trajectory. The machine learning model to be trained is unsupervised by traversing each target object in the current image and at least based on the similarity between the target object and the first positive sample, and the similarity between the target object and the first negative sample.

2. The method according to claim 1, wherein, Based on the representation vectors corresponding to at least one target object in the image, determine the identifiers corresponding to at least one target object in the image, including: If the image is not the first image in the sample video to contain the target object, then the identifier corresponding to at least one target object in the image is determined based on the representation vector corresponding to at least one target object in the image and the identifier corresponding to at least one target object in the historical images preceding the image. Based on the identifiers corresponding to at least one target object in the image and the identifiers corresponding to at least one target object in previous historical images, the uncertainty of the identifier corresponding to each target object in the image is calculated. The uncertainty is inversely proportional to the similarity between target objects with the same identifier and directly proportional to the similarity between target objects with different identifiers. Correct the identifiers in the image whose uncertainty is greater than or equal to a preset threshold.

3. The method according to claim 2, wherein, Based on the identifiers corresponding to at least one target object in the image and the identifiers corresponding to at least one target object in previous historical images, the uncertainty of the identifier corresponding to each target object in the image is calculated, including: Based on the identifier corresponding to any target object in the image, determine a target object in a previous historical image that has the same identifier as the target object, and multiple target objects in the historical image that have different identifiers than the target object; The uncertainty of the identifier corresponding to any target object is calculated based on the similarity between the target object and any target object, and the maximum value among the similarity between the multiple target objects and any target object.

4. The method according to claim 1, wherein, Based on the historical motion trajectory of each target object in the current image, a mapped image of the current image is determined, including: Based on the uncertainty of the corresponding identifier of the target object that meets the preset conditions in the at least one historical image, a target historical image is determined from the at least one historical image, wherein the uncertainty of the corresponding identifier of the target object that meets the preset conditions in the target historical image is the largest. Based on the current image and the target historical image, determine the mapping image of the current image.

5. The method according to claim 4, wherein, Determining the mapped image of the current image based on the current image and the target historical image includes: Based on the location information of the target object that meets the preset conditions in the current image and the location information of the target object that meets the preset conditions in the target historical image, the mapping relationship between the current image and the target historical image is determined; Based on the mapping relationship, the current image is mapped to obtain the mapped image of the current image.

6. The method according to claim 4, wherein, Unsupervised training of the machine learning model to be trained is performed based at least on the similarity between the target object and the first positive sample, and the similarity between the target object and the first negative sample, including: Determine a second positive sample and a second negative sample corresponding to the target object. The second positive sample is a target object in the target historical image that has the same identifier as the target object, or the second positive sample is a target object randomly selected from a first set, the first set including at least one target object in the historical image that has the same identifier as the target object. The second negative sample is a target object in the target historical image that has a different identifier than the target object. A loss function is constructed based at least on the similarity between the target object and the first positive sample, the similarity between the target object and the second positive sample, the similarity between the target object and the first negative sample, and the similarity between the target object and the second negative sample; The machine learning model to be trained is subjected to unsupervised training based on the loss function.

7. The method according to claim 6, wherein, After determining the second positive sample and the second negative sample corresponding to the target object, the method further includes: Determine a third negative sample corresponding to the target object. The third negative sample is at least one target object in a second set. The second set includes at least one of the following: target objects with different identifiers corresponding to the target object in historical images other than the target historical image in the at least one historical image; target objects with different identifiers corresponding to the target object in the mapping images corresponding to the at least one historical image; target objects with different identifiers corresponding to the target object in the current image; and target objects appearing in other videos different from the sample video. A loss function is constructed based at least on the similarity between the target object and the first positive sample, the similarity between the target object and the second positive sample, the similarity between the target object and the first negative sample, and the similarity between the target object and the second negative sample, including: A loss function is constructed based on the similarity between the target object and the first positive sample, the target object and the second positive sample, the target object and the first negative sample, the target object and the second negative sample, and the target object and the third negative sample.

8. A video processing method, wherein, The method includes: Acquire a target video, which includes multiple frames of images; For each frame of the multi-frame images, at least one target region in the image is encoded by a machine learning model to obtain a representation vector corresponding to at least one target object in the image. The target region includes one or more target objects. The machine learning model is trained by the method described in any one of claims 1-7. Based on the representation vectors corresponding to at least one target object in the image, determine the identifiers corresponding to at least one target object in the image; Based on the identifiers corresponding to at least one target object in the image and the identifiers corresponding to at least one target object in previous historical images, the uncertainty of the identifier corresponding to each target object in the image is calculated. The uncertainty is inversely proportional to the similarity between target objects with the same identifier and directly proportional to the similarity between target objects with different identifiers. Correct the identifiers in the image whose uncertainty is greater than or equal to a preset threshold.

9. A video processing method, wherein, The method includes: Receive vehicle video and acquire multiple frames of images from the vehicle video; For each frame of the multi-frame images, at least one target region in the image is encoded by a machine learning model to obtain a representation vector corresponding to at least one vehicle in the image. The target region includes one or more vehicles. The machine learning model is trained by the method described in any one of claims 1-7. Based on the representation vectors corresponding to at least one vehicle in the image, determine the identifiers corresponding to at least one vehicle in the image; Based on the identifiers corresponding to at least one vehicle in the image and the identifiers corresponding to at least one vehicle in previous historical images, the uncertainty of the identifier corresponding to each vehicle in the image is calculated. The uncertainty is inversely proportional to the similarity between target objects with the same identifier and directly proportional to the similarity between target objects with different identifiers. Correct the identifiers in the image whose uncertainty is greater than or equal to a preset threshold.

10. A video processing method, wherein, The method includes: Receive a video of a person and extract multiple frames of images from the video of the person; For each frame of the multi-frame images, at least one target region in the image is encoded by a machine learning model to obtain a representation vector corresponding to at least one person in the image. The target region includes one or more people. The machine learning model is trained by the method described in any one of claims 1-7. Based on the representation vectors corresponding to at least one person in the image, determine the identifiers corresponding to at least one person in the image; Based on the identifiers corresponding to at least one person in the image and the identifiers corresponding to at least one person in previous historical images, the uncertainty of the identifier corresponding to each person in the image is calculated. The uncertainty is inversely proportional to the similarity between target objects with the same identifier and directly proportional to the similarity between target objects with different identifiers. Correct the identifiers in the image whose uncertainty is greater than or equal to a preset threshold.

11. A machine learning model training device, wherein, include: The encoding module is used to encode at least one target region in each frame of the sample video using a machine learning model to be trained, so as to obtain a representation vector corresponding to at least one target object in the image, wherein the target region includes one or more target objects. The first determining module is used to determine the identifiers corresponding to at least one target object in the image based on the representation vectors corresponding to at least one target object in the image. The second determining module is used to select a current image from the sample video, determine the historical motion trajectory of each target object in the current image based on the identifiers corresponding to at least one target object in the current image and the identifiers corresponding to target objects in at least one previous historical image, and determine a mapping image of the current image based on the historical motion trajectories of each target object in the current image. The mapping image of the current image includes a first positive sample and a first negative sample corresponding to the target object with the highest identifier probability in the current image. The target object with the highest identifier probability is determined from the current image based on the historical motion trajectory of each target object in the current image, and the target object that meets the preset conditions has the highest probability of corresponding to the same identifier in its historical motion trajectory. The training module is used to traverse each target object in the current image and perform unsupervised training on the machine learning model to be trained, based at least on the similarity between the target object and the first positive sample, and the similarity between the target object and the first negative sample.

12. An electronic device, wherein, include: Memory; processor; as well as Computer programs; The computer program is stored in the memory and configured to be executed by the processor to implement the method as described in any one of claims 1-10.

13. A computer-readable storage medium having a computer program stored thereon, wherein, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-10.

Images