Object path information determination method and apparatus, device, storage medium, and product

Abstract

The present disclosure relates to a method, apparatus, device, storage medium and product for determining object path information, the method comprising: obtaining at least one image group corresponding to a video stream; determining an object candidate region corresponding to at least two images and an object recognition result corresponding to the object candidate region; if the object recognition result indicates that the object candidate region corresponding to the at least two images includes a same target object, determining position information of the target object in the at least two images; and determining path information corresponding to the target object based on the position information. The embodiments of the present disclosure can determine the same object in continuous images through the object candidate region across images, reduce the complexity and time consumption of object recognition association between different images, and improve the accuracy of object recognition association, thereby improving the efficiency and accuracy of object path determination as a whole.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer, and particularly relates to a method and device for determining object path information, equipment, storage medium and product. BACKGROUND

[0002] Multi-object path determination technology needs to identify objects appearing in a video and give complete target trajectories, while distinguishing different objects. This technology has a wide range of applications in video analysis, autonomous driving, robot control and other fields.

[0003] In the related art, a method of detecting first and then associating is used to determine the paths of multiple objects in a video. The related art needs to detect each frame of picture in the video to obtain objects in each frame of picture, and then associate the same objects detected in different frames to obtain object paths.

[0004] The related art has high complexity and long time consumption in determining object paths. SUMMARY

[0005] The present disclosure provides a method and device for determining object path information, equipment, storage medium and product to at least solve the problem of high complexity and long time consumption in determining object paths in the related art. The technical solutions of the present disclosure are as follows:

[0006] According to a first aspect of an embodiment of the present disclosure, a method for determining object path information is provided, comprising:

[0007] Obtaining at least one image group corresponding to a video stream, any image group comprising at least two consecutive frames of images in the video stream;

[0008] Determining object candidate regions corresponding to the at least two frames of images and object recognition results corresponding to the object candidate regions, the object recognition results representing object information of objects contained in the object candidate regions;

[0009] If the object recognition results indicate that the object candidate regions corresponding to the at least two frames of images include the same target object, determining position information of the target object in the at least two frames of images;

[0010] Based on the position information, determining path information corresponding to the target object, the path information representing a motion trajectory of the target object in the video stream.

[0011] In one possible design, the determining of the object candidate regions corresponding to the at least two frames of images and the object recognition results corresponding to the object candidate regions comprises:

[0012] perform feature extraction processing on the at least two images to obtain image features corresponding to the at least two images;

[0013] perform a candidate box generation operation according to the image features to obtain the object candidate region;

[0014] obtain an object candidate region set based on the object candidate region;

[0015] perform object recognition processing on the at least two images based on a target object candidate region to obtain the object recognition result, the target object candidate region being an object candidate region to be processed in the object candidate region set.

[0016] In one possible design, the at least two images include a first image and a second image, the first image and the second image being adjacent images in the at least two images, and the image features include a first feature map corresponding to the first image and a second feature map corresponding to the second image;

[0017] The object recognition processing on the at least two images based on the target object candidate region to obtain the object recognition result includes:

[0018] perform fusion processing on the first feature map and the second feature map to obtain a fused feature map;

[0019] perform object feature extraction operation on the fused feature map based on the target object candidate region to obtain object features, the object features representing object information associated with the target object candidate region;

[0020] perform object detection operation according to the object features to obtain the object recognition result.

[0021] In one possible design, the object detection operation according to the object features to obtain the object recognition result includes:

[0022] determine a target image and a verification image corresponding to the fused feature map, a fusion order of a feature map corresponding to the target image being before a fusion order of a feature map corresponding to the verification image;

[0023] perform object classification processing based on the object features to obtain first object category information corresponding to a target object candidate region in the target image and second object category information corresponding to a target object candidate region in the verification image;

[0024] perform object identification processing based on the object features to obtain an object identification score, the object identification score representing a homology degree between an object corresponding to the first object category information and an object corresponding to the second object category information;

[0025] perform region regression processing on the target object candidate region based on the object feature, to obtain an object position region corresponding to the target image;

[0026] The object recognition result includes the first object category information, second object category information, object discrimination score, and the object position region.

[0027] In one possible design, if the object recognition result indicates that the object candidate regions corresponding to the at least two images include the same target object, the method further includes determining position information of the target object in the at least two images, including:

[0028] If the first object category information is the same as the second object category information, and the object discrimination score is higher than a discrimination score threshold, it is determined that the target object candidate region corresponding to the target image and the target object candidate region corresponding to the verification image include the same target object.

[0029] The object position region is determined as the position information of the target object in the target image.

[0030] In one possible design, the fusion feature map includes a first fusion feature map corresponding to the first image and a second fusion feature map corresponding to the second image, the target image corresponding to the first fusion feature map is the first image, the verification image corresponding to the first fusion feature map is the second image, the target image corresponding to the second fusion feature map is the second image, and the verification image corresponding to the second fusion feature map is the first image.

[0031] In one possible design, the object feature is obtained by performing an object feature extraction operation on the fusion feature map based on the object candidate region, including:

[0032] determining a feature extraction region corresponding to the object candidate region;

[0033] performing a transformation operation on the feature extraction region to obtain a transformed feature extraction region;

[0034] performing an object feature extraction operation on the fusion feature map based on the transformed feature extraction region to obtain the object feature.

[0035] In one possible design, the method further includes:

[0036] determining a previous image group corresponding to an image group corresponding to the at least two images;

[0037] obtaining a position region corresponding to an object in the previous image group;

[0038] The object candidate region set is obtained based on the object candidate region.

[0039] The object candidate region set is obtained based on the object candidate region and the position region.

[0040] In one possible design, the determining of the object candidate region corresponding to the at least two frames of images and the object recognition result corresponding to the object candidate region includes:

[0041] The object recognition model is input with the at least two frames of images, to output the object candidate region and the object recognition result.

[0042] The object recognition model is a machine learning model trained based on a sample video and label information, the sample video includes at least one sample image group, any sample image group includes at least two sample images in the sample video in sequence, and the label information includes an object position annotation region corresponding to the at least two sample images, object identification information, and object category annotation information corresponding to the object identification information.

[0043] In one possible design, the training process of the object recognition model includes:

[0044] The object recognition model to be trained is input with the at least two sample images, to obtain a sample object candidate region corresponding to the at least two sample images, sample object category information, a sample object discrimination score, and a sample object position region.

[0045] A region coverage between the sample object candidate region and the object position annotation region is determined.

[0046] The sample object candidate region is classified based on the region coverage, to obtain a positive sample object candidate region and a negative sample object candidate region.

[0047] First loss information is determined based on the sample object position region and the object position annotation region, the first loss information representing a difference degree between the sample object position region and the object position annotation region.

[0048] Second loss information is determined based on the sample object category information and the object category annotation information, the second loss information representing a difference degree between the sample object category information and the object category annotation information.

[0049] Third loss information is determined based on the sample object discrimination score and the object identification information, the third loss information representing a correctness degree of the sample object discrimination score.

[0050] According to the positive sample object candidate region, the negative sample object candidate region, the first loss information, the second loss information, and the third loss information, a model parameter updating process is performed to obtain the object recognition model.

[0051] In a possible design, the at least two sample images include a first sample image and a second sample image, the first sample image and the second sample image are adjacent images, the object position annotation region includes a first annotation region corresponding to the first sample image and a second annotation region corresponding to the second sample image, and the determination of the area coverage between the sample object candidate region and the object position annotation region includes:

[0052] determination of a first coverage between the sample object candidate region and the first annotation region and a second coverage between the sample object candidate region and the second annotation region, the area coverage including the first coverage and the second coverage;

[0053] the classification of the sample object candidate region based on the area coverage to obtain a positive sample object candidate region and a negative sample object candidate region, including:

[0054] if the first coverage is greater than a coverage threshold and the second coverage is greater than the coverage threshold, the sample object candidate region is determined as the positive sample object candidate region;

[0055] if the first coverage is less than or equal to the coverage threshold or the second coverage is less than or equal to the coverage threshold, the sample object candidate region is determined as the negative sample object candidate region.

[0056] According to a second aspect of the embodiments of the present disclosure, an object path information determination apparatus is provided, including:

[0057] an image group acquisition module configured to perform acquisition of at least one image group corresponding to a video stream, any image group including at least two consecutive frames of images in the video stream;

[0058] an object recognition module configured to perform determination of an object candidate region corresponding to the at least two frames of images and an object recognition result corresponding to the object candidate region, the object recognition result representing object information of an object included in the object candidate region;

[0059] a position determination module configured to perform determination of position information of a target object in the at least two frames of images if the object recognition result indicates that the object candidate region corresponding to the at least two frames of images includes the same target object;

[0060] A path determination module configured to determine path information corresponding to the target object based on the position information, the path information representing a motion trajectory of the target object in the video stream.

[0061] In one possible design, the object identification module includes:

[0062] An image feature extraction submodule configured to perform feature extraction processing on the at least two images to obtain image features corresponding to the at least two images.

[0063] A candidate region determination submodule configured to perform a candidate box generation operation according to the image features to obtain the object candidate regions.

[0064] A region set determination submodule configured to obtain a set of object candidate regions based on the object candidate regions.

[0065] An object identification submodule configured to perform object identification processing on the at least two images based on a target object candidate region to obtain the object identification result, the target object candidate region being an object candidate region to be processed in the set of object candidate regions.

[0066] In one possible design, the at least two images include a first image and a second image, the first image and the second image being adjacent images in the at least two images, and the image features include a first feature map corresponding to the first image and a second feature map corresponding to the second image.

[0067] The object identification submodule includes:

[0068] A feature map fusion unit configured to perform fusion processing on the first feature map and the second feature map to obtain a fused feature map.

[0069] An object feature extraction unit configured to perform object feature extraction on the fused feature map based on the target object candidate region to obtain an object feature, the object feature representing object information associated with the target object candidate region.

[0070] An object feature detection unit configured to perform object detection on the object feature to obtain the object identification result.

[0071] In one possible design, the object feature detection unit includes:

[0072] An image determination subunit configured to determine a target image and a verification image corresponding to the fused feature map, a fusion order of a feature map corresponding to the target image being earlier than a fusion order of a feature map corresponding to the verification image.

[0073] an object classification subunit, configured to perform object classification processing based on the object feature, to obtain first object category information corresponding to a target object candidate region in the target image and second object category information corresponding to a target object candidate region in the verification image;

[0074] an object identification subunit, configured to perform object identification processing based on the object feature, to obtain an object identification score, the object identification score representing a homology degree between an object corresponding to the first object category information and an object corresponding to the second object category information;

[0075] a region regression subunit, configured to perform region regression processing on the target object candidate region based on the object feature, to obtain an object position region corresponding to the target image;

[0076] wherein the object recognition result comprises the first object category information, the second object category information, the object identification score, and the object position region.

[0077] In one possible design, the position determination module comprises:

[0078] a target object determination unit, configured to perform, if the first object category information is the same as the second object category information and the object identification score is higher than an identification score threshold, determination that the target object candidate region corresponding to the target image and the target object candidate region corresponding to the verification image comprise a same target object;

[0079] a position region determination unit, configured to perform determination of the object position region as position information of the target object in the target image.

[0080] In one possible design, the fused feature map comprises a first fused feature map corresponding to the first image and a second fused feature map corresponding to the second image, the target image corresponding to the first fused feature map is the first image, the verification image corresponding to the first fused feature map is the second image, the target image corresponding to the second fused feature map is the second image, and the verification image corresponding to the second fused feature map is the first image.

[0081] In one possible design, the object feature extraction unit comprises:

[0082] an extraction region determination subunit, configured to perform determination of a feature extraction region corresponding to the object candidate region;

[0083] an extraction region transformation subunit, configured to perform a transformation operation on the feature extraction region, to obtain a transformed feature extraction region.

[0084] An object feature extraction subunit is configured to perform an object feature extraction operation on the fused feature map based on the transformed feature extraction region to obtain the object feature.

[0085] In a possible design, the apparatus further includes:

[0086] An image group determination module is configured to perform determination of a previous image group corresponding to an image group to which the at least two images belong.

[0087] A position region acquisition module is configured to perform acquisition of a position region corresponding to an object in the previous image group.

[0088] The region set determination sub-module is further configured to perform obtaining of the object candidate region set based on the object candidate region and the position region.

[0089] In a possible design, the object recognition module is further configured to perform inputting of the at least two images into an object recognition model to output the object candidate region and the object recognition result.

[0090] The object recognition model is a machine learning model trained based on a sample video and label information, the sample video includes at least one sample image group, any sample image group includes at least two sample images in the sample video in sequence, and the label information includes an object position annotation region corresponding to the at least two sample images, object identification information, and object category annotation information corresponding to the object identification information.

[0091] In a possible design, the apparatus further includes a model training apparatus, and the model training apparatus includes:

[0092] A sample object recognition module is configured to perform inputting of the at least two sample images into a to-be-trained object recognition model to obtain a sample object candidate region, sample object category information, a sample object discrimination score, and a sample object position region corresponding to the at least two sample images.

[0093] A coverage rate determination module is configured to perform determination of a region coverage rate between the sample object candidate region and the object position annotation region.

[0094] A candidate region classification module is configured to perform classification processing of the sample object candidate region based on the region coverage rate to obtain a positive sample object candidate region and a negative sample object candidate region.

[0095] The first loss information determination module is configured to determine first loss information based on the sample object location region and the object location annotation region, the first loss information representing a difference degree between the sample object location region and the object location annotation region.

[0096] The second loss information determination module is configured to determine second loss information based on the sample object category information and the object category annotation information, the second loss information representing a difference degree between the sample object category information and the object category annotation information.

[0097] The third loss information determination module is configured to determine third loss information based on the sample object discrimination score and the object identification information, the third loss information representing a correctness degree of the sample object discrimination score.

[0098] The model parameter updating module is configured to perform model parameter updating processing according to the positive sample object candidate region, the negative sample object candidate region, the first loss information, the second loss information, and the third loss information, to obtain the object recognition model.

[0099] In a possible design, the at least two sample images include a first sample image and a second sample image, the first sample image and the second sample image being adjacent images, and the object location annotation region includes a first annotation region corresponding to the first sample image and a second annotation region corresponding to the second sample image.

[0100] The coverage determination module is specifically configured to determine a first coverage between the sample object candidate region and the first annotation region, and a second coverage between the sample object candidate region and the second annotation region, the region coverage including the first coverage and the second coverage.

[0101] The candidate region classification module includes:

[0102] The positive sample determination unit is configured to determine the sample object candidate region as the positive sample object candidate region if the first coverage is greater than a coverage threshold and the second coverage is greater than the coverage threshold.

[0103] The negative sample determination unit is configured to determine the sample object candidate region as the negative sample object candidate region if the first coverage is less than or equal to the coverage threshold or the second coverage is less than or equal to the coverage threshold.

[0104] According to a third aspect of the embodiments of the present disclosure, an electronic device is provided, comprising: a processor; a memory configured to execute instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the method according to any one of the first aspect.

[0105] According to a fourth aspect of the embodiments of the present disclosure, a computer readable storage medium is provided, when instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method according to any one of the first aspect of the embodiments of the present disclosure.

[0106] According to a fifth aspect of the embodiments of the present disclosure, a computer program product containing instructions is provided, when the computer program product is executed on a computer, the computer is enabled to execute the method according to any one of the first aspect of the embodiments of the present disclosure.

[0107] The embodiments of the present disclosure provide at least the following beneficial effects:

[0108] By determining the object candidate region corresponding to the at least two continuous frames of images in the video stream, the object recognition result corresponding to the object candidate region can be determined, and by the object recognition result, it can be determined whether the object candidate region corresponding to the at least two frames of images includes the same target object, if the determination result is yes, the same object in the continuous images is determined through the object candidate region across the images, the complexity and time consumption of object recognition association between different images are reduced, and the accuracy of object recognition association is improved, after it is determined that the object candidate region corresponding to the continuous images includes the same target object, the position information of the target object in the at least two frames of images can be determined through the above object recognition result, and then the path information of the target object in the video stream is determined, the efficiency and accuracy of object path determination are improved as a whole.

[0109] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0110] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the present disclosure and, together with the specification, serve to explain the principles of the present disclosure, and do not constitute an improper limitation on the present disclosure.

[0111] Figure 1 is a schematic diagram of an application environment according to an exemplary embodiment.

[0112] Figure 2 is a flowchart of a method for determining object path information according to an exemplary embodiment. Figure One ;

[0113] Figure 3 is a flow of a method for determining object path information according to an example embodiment Figure Two ;

[0114] Figure 4 is a flow of a method for determining object path information according to an example embodiment Figure Three ;

[0115] Figure 5 is a flow of a method for determining object path information according to an example embodiment Figure Four ;

[0116] Figure 6 is a flowchart of a method for training an object recognition model according to an example embodiment

[0117] Figure 7 an example illustrates a flowchart of determining path information based on an object recognition model

[0118] Figure 8 is a block diagram of a device for determining object path information according to an example embodiment

[0119] Figure 9 is a block diagram of an electronic device for determining object path information according to an example embodiment DETAILED DESCRIPTION

[0120] In order to make the ordinary people in the art better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be described clearly and completely below in conjunction with the drawings.

[0121] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily have to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation described in the following example embodiments does not represent all implementations consistent with the present disclosure. Rather, they are only examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0122] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for display, analyzed data, etc.) involved in the present disclosure are all information and data authorized by the user or authorized by all parties.

[0123] Please refer to Figure 1 ,Figure 1 is a schematic diagram of an application environment according to an exemplary embodiment, as Figure 1 shown, the application environment can include a terminal 100 and a server 200.

[0124] The terminal 100 can be used to provide an object path determination service for any user. Specifically, the terminal 100 includes, but is not limited to, electronic devices such as smart phones, desktop computers, tablet computers, notebook computers, vehicle-mounted terminals, aircraft, ship terminals, augmented reality (AR) / virtual reality (VR) devices, smart wearable devices, etc. Optionally, the terminal 100 is installed with software that provides the above-mentioned object path determination service, such as an application program, etc. Optionally, the operating system running on the electronic device can include, but is not limited to, Android system, IOS system, Linux, Windows, etc.

[0125] In an optional embodiment, the server 200 can provide background services for the terminal 100. Specifically, the server 200 can be a standalone physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms, etc. Basic cloud computing services. Optionally, the server 200 is installed with software that provides the above-mentioned object path determination service, such as an application program, etc.

[0126] In addition, it should be noted that Figure 1 shown is only an application environment provided by the present disclosure, and in actual application, it can also include other application environments, for example, it can include more terminals.

[0127] In the embodiments of the present disclosure, the above-mentioned terminal 100 and server 200 can be directly or indirectly connected through wired or wireless communication, which is not limited by the present disclosure.

[0128] Figure 2 is a flowchart of a method for determining object path information according to an exemplary embodiment Figure One , as Figure 2 shown, the method for determining object path information is applied in an electronic device, such as a terminal or a server, and the above-mentioned method includes the following steps (210-240).

[0129] Step 210, at least one image group corresponding to a video stream is obtained.

[0130] Optionally, the any image group comprises at least two consecutive images in the video stream. The source of the video stream is not limited in the embodiments of the present disclosure.

[0131] Optionally, the image group comprises two adjacent images in the video stream.

[0132] In step 220, the object candidate region corresponding to the at least two images and the object recognition result corresponding to the object candidate region are determined.

[0133] The embodiments of the present disclosure are based on two assumptions. The first assumption is that the same object has similar positions in adjacent frames. The second assumption is that the appearance of the same object changes little in adjacent frames. Thus, the object candidate region, such as a reference rectangular frame, can be used to compare the features in adjacent frames to determine whether the objects in the two frames belong to the same object and predict the position coordinates of the object in the two frames.

[0134] Optionally, the object candidate region is an object candidate region corresponding to the at least two images, such as an object candidate region corresponding to at least two images. Optionally, the object candidate region comprises an object candidate frame across image frames, which is used to connect the same object in different images in time sequence in the matching video stream. For example, the same object in adjacent image frames is connected through the object candidate frame across image frames.

[0135] Optionally, the object candidate region is represented by an object candidate frame (Proposal). According to the object candidate region, the region images corresponding to the object candidate region in the at least two images can be determined.

[0136] Optionally, the object recognition result represents the object information of the object contained in the object candidate region. Optionally, the object recognition result comprises the object recognition information of the region images corresponding to the object candidate region in the at least two images. For example, the object category information corresponding to each region image, the object discrimination score corresponding to the object category information, and the object position region corresponding to each region image.

[0137] Optionally, the object discrimination score represents the homology degree between the objects corresponding to the object category information of each region image.

[0138] In the exemplary embodiments, the step 220 can be performed by a trained object recognition model. Optionally, the at least two images are input into the object recognition model, and the object candidate region and the object recognition result are output.

[0139] The object recognition model is a machine learning model trained on sample videos and label information. The sample videos include at least one group of sample images, and any group of sample images includes at least two consecutive sample images in the sample videos. The label information includes the object location annotation region, object identification information, and object category annotation information corresponding to the object identification information for at least two sample images.

[0140] Optionally, the object recognition model includes a backbone network, a region proposal network (RPN), a proposal pool, an object feature extraction network (such as a deformable feature extraction network), and a detection branch network. Optionally, at least two frames of images input to the object recognition model are input image frames (query frames).

[0141] In practical applications, the aforementioned backbone network can adopt high-performance structures such as ResNet (Residual Network) 50, ResNet 101, ResNet 200, ResNeXt 101, and ResNeSt 101, or faster structures such as ResNet 18, MobileNet (Lightweight Network), and SqueezeNet (Compression Strategy Network). This disclosure does not limit the network structure of the backbone network.

[0142] Optionally, the detection branch network described above is implemented based on a Region Convolutional Neural Network (RCNN). Optionally, the detection branch network described above is an RCNN head. This disclosure does not limit the network structure of the detection branch network.

[0143] Optionally, the sample dataset corresponding to the above object recognition model includes the ImageNet dataset and the MS COCO dataset, and this disclosure does not limit this.

[0144] The trained object recognition model described above can identify object recognition information in the object candidate regions of each image in a continuous series of input images, achieving end-to-end object recognition and improving object recognition efficiency and accuracy.

[0145] In an exemplary embodiment, such as Figure 3 As shown, after inputting at least two frames of images into the object recognition model, the execution process corresponding to the object recognition model, i.e., the execution process of step 220, may include the following steps (221-224). Figure 3Fig. 1 is a flowchart of a method for determining object path information according to an example embodiment Figure Two .

[0146] At step 221, feature extraction processing is performed on the at least two images to obtain image features corresponding to the at least two images.

[0147] In a possible implementation, the feature extraction processing is performed on the at least two images based on the backbone network to obtain the image features corresponding to the at least two images.

[0148] Optionally, the at least two images include a first image and a second image, and the first image and the second image are adjacent images in the at least two images. Correspondingly, the image features include a first feature map corresponding to the first image and a second feature map corresponding to the second image.

[0149] Optionally, the first image is input into the backbone network in the object recognition model, and the feature extraction processing is performed on the first image based on the backbone network to obtain the first feature map.

[0150] Optionally, the second image is input into the backbone network in the object recognition model, and the feature extraction processing is performed on the second image based on the backbone network to obtain the second feature map.

[0151] Optionally, a repeated image between the current image group and the next image group is determined, and an image feature corresponding to the repeated image is retained to be reused in an object recognition process corresponding to the next image group. For example, when images A and B are calculated, an image feature F_b of the image B is retained and used in object recognition calculation of the image B and the image C.

[0152] At step 222, a candidate box generation operation is performed according to the image features to obtain an object candidate region.

[0153] In a possible implementation, the image features are input into the candidate box generation network, and the candidate box generation network performs the candidate box generation operation according to the image features to obtain object candidate boxes corresponding to the at least two images respectively, and the object candidate boxes represent the object candidate regions.

[0154] Optionally, the first feature map is input into the candidate box generation network, and the candidate box generation network performs the candidate box generation operation according to the first feature map to obtain a first object candidate box corresponding to the first image. The first object candidate box includes an object candidate box corresponding to a first object in the first image. Optionally, the first object is an object in the first image.

[0155] Optionally, the second feature map is input into the candidate box generation network, and the candidate box generation network performs a candidate box generation operation according to the second feature map to obtain a second object candidate box corresponding to the second image. The second object candidate box includes an object candidate box corresponding to a second object in the second image. Optionally, the second object is an object in the second image.

[0156] In step 223, the object candidate region set is obtained based on the object candidate region.

[0157] In a possible implementation, the object candidate boxes corresponding to the at least two images are stored in a candidate box pool. The candidate box pool represents the object candidate region set.

[0158] Optionally, the first object candidate box and the second object candidate box are stored in the candidate box pool.

[0159] In another possible implementation, a previous image group corresponding to an image group to which the at least two images belong can also be determined, and a position region corresponding to an object in the previous image group can be obtained. Accordingly, the object candidate region set is obtained based on the object candidate region and the position region.

[0160] Optionally, a current image group to which the at least two images input this time belong is determined, and a previous image group corresponding to the current image group is determined.

[0161] Optionally, the position region includes an object position box corresponding to the at least two images in the previous image group.

[0162] Optionally, the position region further includes an object candidate box corresponding to a repeated image between the previous image group and the current image group.

[0163] Optionally, the first object candidate box, the second object candidate box, the object position box corresponding to the at least two images in the previous image group, and the object candidate box corresponding to the repeated image are stored in the candidate box pool.

[0164] For example, the object recognition model obtains a position rectangular box Box_a and Box_b of a target object from the initial two images A and B, and then stores the Box_a and Box_b in a candidate box pool corresponding to the image B and the image C.

[0165] By storing the position region corresponding to the previous image group in the object candidate region set, the richness and accuracy of the object candidate region can be improved.

[0166] In step 224, object recognition processing is performed on the at least two images based on the target object candidate region to obtain an object recognition result.

[0167] Optionally, the target object candidate region is an object candidate region to be processed in the set of object candidate regions.

[0168] By extracting the image features of the at least two images, the object candidate regions corresponding to the respective images can be determined and the set of object candidate regions can be obtained. Then, the object recognition can be performed on the image content corresponding to each object candidate region in the set in the respective images to obtain the object recognition result corresponding to each object candidate region, so as to determine the object candidate regions including the same object in the continuous images in the candidate process, and the accuracy of the object recognition is effectively improved.

[0169] In an example embodiment, as shown in FIG. 22, Figure 4 The implementation process of the step 224 includes the following steps (2241-2243), Figure 4 is a flow of a method for determining object path information according to an example embodiment Figure Three .

[0170] In step 2241, the first feature map and the second feature map are fused to obtain a fused feature map.

[0171] Optionally, the fused feature map includes a first fused feature map corresponding to the first image and a second fused feature map corresponding to the second image. The first fused feature map and the second fused feature map correspond to different feature map fusion orders. The feature map fusion order is used to distinguish the target image and the verification image corresponding to the fused feature map. The fusion order of the feature map corresponding to the target image is before the feature map corresponding to the verification image. The target image is the detection object corresponding to the fused feature map, and the verification image is used to verify the target image.

[0172] In a possible implementation, the first feature map and the second feature map are processed in parallel according to the order from the first feature map to the second feature map to obtain the first fused feature map; and the second feature map and the first feature map are processed in parallel according to the order from the second feature map to the first feature map to obtain the second fused feature map.

[0173] The target image corresponding to the first fused feature map is the first image, and the verification image corresponding to the first fused feature map is the second image; the target image corresponding to the second fused feature map is the second image, and the verification image corresponding to the second fused feature map is the first image.

[0174] By different feature fusion orders, the fused feature maps corresponding to different images in the continuous images can be determined, and the feature fusion order of each feature fusion map determines the target image corresponding to the fused feature map, so as to distinguish different images in the continuous images, and the accuracy of the object recognition is improved.

[0175] At step 2242, object feature extraction is performed on the fused feature map based on the target object candidate region, to obtain object features.

[0176] Optionally, the object features represent object information associated with the target object candidate region. The object information associated with the target object candidate region includes object information in the target object candidate region corresponding to at least two images, such as object information in the target object candidate region corresponding to the first image and object information in the target object candidate region corresponding to the second image.

[0177] Optionally, a feature extraction region corresponding to the target object candidate region is determined, and object feature extraction is performed on the fused feature map based on the feature extraction region, to obtain the object features.

[0178] In one possible implementation, object feature extraction is performed on the first fused feature map based on a feature extraction region corresponding to the target object candidate region, to obtain first object features representing object information in the feature extraction region corresponding to the first fused feature map; and object feature extraction is performed on the second fused feature map based on the feature extraction region corresponding to the target object candidate region, to obtain second object features representing object information in the feature extraction region corresponding to the second fused feature map.

[0179] Optionally, the object feature extraction is performed based on an object feature extraction network in the object recognition model.

[0180] In another possible implementation, the object feature extraction is deformable feature extraction, and the specific execution process includes: determining a feature extraction region corresponding to the object candidate region; performing a transformation operation on the feature extraction region to obtain a transformed feature extraction region; and performing object feature extraction on the fused feature map based on the transformed feature extraction region, to obtain the object features.

[0181] Optionally, object feature extraction is performed on the first fused feature map based on the transformed feature extraction region, to obtain expanded first object features representing object information in the transformed feature extraction region corresponding to the first fused feature map; and object feature extraction is performed on the second fused feature map based on the transformed feature extraction region, to obtain expanded second object features representing object information in the transformed feature extraction region corresponding to the second fused feature map.

[0182] Optionally, the deformable feature extraction is performed based on a deformable feature extraction network in the object recognition model.

[0183] Through the deformable feature extraction operation, the features around the target object candidate region can be extracted, so as to expand the coverage of the target object candidate region to adapt to a larger object offset, so as to associate the objects with a larger offset in the continuous images, and the robustness of the object candidate box across the images to the object offset in the continuous images is improved.

[0184] In step 2243, an object detection operation is performed according to the object feature, and an object recognition result is obtained.

[0185] Optionally, the object feature is input into the detection branch network, and the detection branch network performs an object detection operation according to the input object feature to obtain an object recognition result.

[0186] In an example embodiment, the object feature includes the first object feature and the second object feature. Accordingly, the object recognition result includes a first object recognition result corresponding to the first object feature and an object recognition result corresponding to the second object feature.

[0187] Optionally, the object detection operation is performed on the first object feature to obtain a first object recognition result corresponding to a region image corresponding to the target object candidate region in the first image. The first object recognition result includes a first object class result, a first object discrimination score, and a first object position region. The first object class result includes object class information corresponding to the target object candidate region in the first image and object class information corresponding to the target object candidate region in the second image. The first object discrimination score represents the homology between the objects corresponding to the two kinds of object class information. The first object position region represents the position region of the object corresponding to the object class information corresponding to the first image in the first image.

[0188] Optionally, the object detection operation is performed on the second object feature to obtain a second object recognition result corresponding to a region image corresponding to the target object candidate region in the second image. The second object recognition result includes a second object class result, a second object discrimination score, and a second object position region. The second object class result includes object class information corresponding to the target object candidate region in the second image and object class information corresponding to the target object candidate region in the first image. The second object discrimination score represents the homology between the objects corresponding to the two kinds of object class information. The second object position region represents the position region of the object corresponding to the object class information corresponding to the second image in the second image.

[0189] By fusing the image features corresponding to the continuous images respectively, the fused feature map corresponding to the continuous images can be obtained. Based on the above object candidate region, the local object feature extraction operation can be performed on the fused feature map, so as to avoid bringing in the object feature information irrelevant to the object candidate region, and then the object feature corresponding to the object candidate region is obtained. By detecting the object feature, the object recognition result corresponding to the object candidate region can be obtained, and the accuracy of object recognition is improved.

[0190] In an example embodiment, the object recognition result includes first object category information, second object category information, object discrimination score and object position region. Correspondingly, as shown in Figure 5 The implementation process of the step 2243 includes the following steps (2243a-2243d), Figure 5 is a flow of a method for determining object path information according to an example embodiment Figure Four .

[0191] Step 2243a, determining the target image and the verification image corresponding to the fused feature map.

[0192] Optionally, the fusion order of the feature map corresponding to the target image is before the fusion order of the feature map corresponding to the verification image.

[0193] Optionally, the feature map fusion order corresponding to the fused feature map is determined, and the target image and the verification image corresponding to the fused feature map are determined according to the fusion order of different feature maps in the fused feature map.

[0194] Optionally, the first feature map corresponding to the fused feature map is determined as the above target image

[0195] In a possible implementation, if the first fused feature map is received, the first image is determined as the target image, and the second image is determined as the verification image; if the second fused feature map is received, the second image is determined as the target image, and the first image is determined as the verification image.

[0196] Step 2243b, performing object classification processing based on the object feature to obtain the first object category information corresponding to the target object candidate region in the target image and the second object category information corresponding to the target object candidate region in the verification image.

[0197] Optionally, the object classification processing is performed based on the first object feature to obtain object category information corresponding to the target object candidate region in the first image and object category information corresponding to the target object candidate region in the second image. The first object category information corresponding to the first object feature is the object category information corresponding to the target object candidate region in the first image. The second object category information corresponding to the first object feature is the object category information corresponding to the target object candidate region in the second image.

[0198] Optionally, the object classification processing is performed based on the second object feature to obtain object category information corresponding to the target object candidate region in the second image and object category information corresponding to the target object candidate region in the first image. The first object category information corresponding to the second object feature is the object category information corresponding to the target object candidate region in the second image. The second object category information corresponding to the second object feature is the object category information corresponding to the target object candidate region in the first image.

[0199] In step 2243c, the object identification processing is performed based on the object feature to obtain an object identification score.

[0200] Optionally, the object identification score represents a homology degree between the object corresponding to the first object category information and the object corresponding to the second object category information.

[0201] Optionally, the object identification processing is performed based on the first object feature to obtain the first object identification score. The first object identification score represents a homology degree between the object corresponding to the object category information corresponding to the second image and the object corresponding to the object category information corresponding to the first image.

[0202] Optionally, the object identification processing is performed based on the second object feature to obtain the second object identification score. The second object identification score represents a homology degree between the object corresponding to the object category information corresponding to the first image and the object corresponding to the object category information corresponding to the second image.

[0203] Optionally, the object identification score includes the first object identification score and the second object identification score.

[0204] In step 2243d, the region regression processing is performed on the target object candidate region based on the object feature to obtain an object position region corresponding to the target image.

[0205] The target object candidate region is corrected based on the object feature to obtain an object position region of the object corresponding to the object feature in the target image.

[0206] Optionally, based on the first object feature, the target object candidate region is subjected to region regression processing to obtain a first object position region corresponding to the first image. Optionally, the first object position region refers to a position region of the object corresponding to the first object feature in the first image.

[0207] Optionally, based on the second object feature, the target object candidate region is subjected to region regression processing to obtain a second object position region corresponding to the second image. Optionally, the second object position region refers to a position region of the object corresponding to the second object feature in the second image.

[0208] The embodiments of the present disclosure perform classification, identification and regression on adjacent images based on the same cross-frame object candidate box, so that the prediction results of the same cross-frame object candidate box on two adjacent frames of images belong to the same object.

[0209] The object feature determined through the foregoing steps can be subjected to object classification, object identification and region regression, to obtain object category information corresponding to the target object candidate region of the target image and the verification image, respectively, object identification scores capable of representing homology between objects corresponding to the two object category information, and a position region of the object corresponding to the object feature in the target image, thereby obtaining an object recognition result which is rich in information and accurate.

[0210] In step 230, if the object recognition result indicates that the object candidate regions corresponding to at least two frames of images include the same target object, the position information of the target object in the at least two frames of images is determined.

[0211] In a possible implementation, whether the object candidate regions corresponding to at least two frames of images include the same target object can be determined by judging whether the first object category information and the second object category information are the same and whether the object identification score is higher than the identification score threshold. Accordingly, as shown in FIG. 2, the implementation process of step 230 includes the following steps (231-232). Figure 5

[0212] In step 231, if the first object category information and the second object category information are the same and the object identification score is higher than the identification score threshold, it is determined that the target object candidate region corresponding to the target image and the target object candidate region corresponding to the verification image include the same target object.

[0213] ​The case that the first object category information is the same as the second object category information means that the target candidate region corresponding to the target image and the target candidate region corresponding to the verification image include objects of the same category. The case that the object discrimination score is higher than the discrimination score threshold further determines that the objects of the same category in the target image and the verification image are the same object. Therefore, in the case that the first object category information is the same as the second object category information and the object discrimination score is higher than the discrimination score threshold, it can be determined that the target object candidate regions corresponding to the target image and the verification image respectively include the same target object.

[0214] In step 232, the object position region is determined as the position information of the target object in the target image.

[0215] In a possible implementation, if the object category information in the first object recognition result is the same and the first object discrimination score is higher than the discrimination score threshold, it is determined that the target object candidate regions corresponding to the first image and the second image respectively include the same target object, and the first object position region in the first object recognition result is determined as the position information of the target object in the first image.

[0216] If the object category information in the second object recognition result is the same and the second object discrimination score is higher than the discrimination score threshold, it is determined that the target object candidate regions corresponding to the second image and the first image respectively include the same target object, and the second object position region in the second object recognition result is determined as the position information of the target object in the second image.

[0217] By judging whether the first object category information is the same as the second object category information and whether the object discrimination score is higher than the discrimination score threshold, it can be determined whether the object candidate regions corresponding to at least two images include the same target object, and further, in the case that the conditions are met, the object position region in the object recognition result is determined as the position information of the target object, so that the position calibration of the same object in continuous images is realized, and the accuracy of object recognition is ensured.

[0218] In a possible implementation, as shown in FIG. 1, Figure 6 The training process of the object recognition model includes the following steps (301-306), Figure 6 FIG. 3 is a flowchart of a training method of an object recognition model according to an example embodiment.

[0219] In step 301, at least two sample images are input into the object recognition model to be trained, to obtain sample object candidate regions, sample object category information, sample object discrimination scores and sample object position regions corresponding to the at least two sample images.

[0220] Optionally, the at least two sample images include a first sample image and a second sample image, the first sample image and the second sample image are adjacent images, and the object position labeling region includes a first labeling region corresponding to the first sample image and a second labeling region corresponding to the second sample image.

[0221] Optionally, the operation of the object recognition model in the training process is similar to the operation in the application process, except that the loss function is not calculated in the application process, and the positive sample and the corresponding recognition result are directly used as the output prediction result.

[0222] In step 302, the area coverage between the sample object candidate region and the object position labeling region is determined.

[0223] In step 303, the sample object candidate region is classified based on the area coverage to obtain a positive sample object candidate region and a negative sample object candidate region.

[0224] In a possible implementation, the sample object candidate region can be classified by the area coverage between the sample object candidate region and the object position labeling region to obtain a positive sample object candidate region and a negative sample object candidate region. The specific process is as follows:

[0225] The first coverage between the sample object candidate region and the first labeling region is determined, and the second coverage between the sample object candidate region and the second labeling region is determined. The area coverage includes the first coverage and the second coverage.

[0226] If the first coverage is greater than the coverage threshold and the second coverage is greater than the coverage threshold, the sample object candidate region is determined as a positive sample object candidate region. If the first coverage is greater than the coverage threshold and the second coverage is greater than the coverage threshold, it can be explained that the sample object candidate region corresponds to the same labeled object in at least two continuous sample images, and therefore the sample object candidate region can be used as a positive sample for training.

[0227] If the first coverage is less than or equal to the coverage threshold or the second coverage is less than or equal to the coverage threshold, the sample object candidate region is determined as a negative sample object candidate region. The case that the first coverage is less than or equal to the coverage threshold or the second coverage is less than or equal to the coverage threshold can indicate that the sample object candidate region does not correspond to the same labeled object in the at least two continuous sample images at the same time, and thus the sample object candidate region can be trained as a negative sample. For example, the sample object candidate region only corresponds to an object in the target sample image, and does not correspond to the same object in the verification sample image; or the sample object candidate region does not correspond to an object in the target sample image, and thus the sample object candidate region can be a positive sample. The target sample image and the verification sample image are similar to the concepts of the target image and the verification image related to the model application side.

[0228] By determining the coverage between the sample object candidate region output in the training process and the labeled region, the object candidate region with good coverage effect can be determined as a positive sample, and the object candidate region with poor coverage effect can be determined as a negative sample, and then the model can be trained through the positive sample and the negative sample, thereby improving the accuracy of the object candidate region determination.

[0229] In step 304, first loss information is determined based on the sample object position region and the object position labeled region.

[0230] Optionally, the first loss information represents the difference between the sample object position region and the object position labeled region.

[0231] Optionally, a region regression loss function is obtained; and the first loss information, i.e., the region regression loss information, is generated based on the sample object position region, the object position labeled region, and the region regression loss function. The region regression loss function supervises the position of the object candidate frame to ensure the accuracy of the region regression processing.

[0232] In step 305, second loss information is determined based on the sample object category information and the object category labeled information.

[0233] Optionally, the second loss information represents the difference between the sample object category information and the object category labeled information.

[0234] Optionally, a classification loss function is obtained; and the second loss information, i.e., the classification loss information, is generated based on the sample object category information, the object category labeled information, and the classification loss function. The classification loss function supervises whether there is an object in the object candidate frame to ensure the accuracy of the object classification.

[0235] In step 306, third loss information is determined based on the sample object discrimination score and the object identification information.

[0236] Optionally, the third loss information represents a correct degree of the sample object discrimination score.

[0237] Optionally, an identification loss function is obtained; and the third loss information, i.e., the identification loss information, is determined based on the sample object discrimination score, the object identification information, and the identification loss function. The identification loss function supervises whether the object candidate region corresponds to the same object in the at least two sample images.

[0238] At step 307, model parameter updating processing is performed according to the positive sample object candidate region, the negative sample object candidate region, the first loss information, the second loss information, and the third loss information, to obtain an object recognition model.

[0239] Optionally, the network is trained based on a gradient descent method of SGD (Stochastic Gradient Descent). Optionally, the entire object recognition model is trained for 50,000 rounds on a multi-target object path information dataset MOT16, an initial learning rate is 0.02, the learning rate is reduced by 10 times at the 30,000th round and the 40,000th round, and then a trained object recognition model is obtained.

[0240] As can be known from the above description of the application process and the training process of the object recognition model, the training and application deployment process of the object recognition model provided by the present disclosure is efficient, which can effectively improve the training efficiency and improve the object path determination effect.

[0241] The object candidate region determined by the object recognition model and the object position annotation region can determine the positive and negative sample object candidate regions, thereby improving the accuracy of the object recognition model in determining the object candidate region; the position region output by the object recognition model and the position annotation region can determine the first loss information, and then the accuracy of the position region output by the object recognition model can be improved according to the first loss information; the object class annotation information and the object class information output by the object recognition model can determine the second loss information, and then the accuracy of the object recognition model in identifying the object class can be improved according to the second loss information; the object discrimination score output by the object recognition model and the object identification information can determine the third loss information, and then the accuracy of the object recognition model in discriminating the same object can be improved according to the third loss information. Through the above training, end-to-end object recognition can be realized, and the precision of the object recognition model can be improved.

[0242] In addition, the object recognition model uses less label information, the association relationship of the same object in different images is determined through the object candidate region, the complexity of object recognition association is low, and the training efficiency is high.

[0243] At step 240, path information corresponding to the target object is determined based on the position information.

[0244] Optionally, the path information represents a motion trajectory of the target object in the video stream.

[0245] Optionally, the object position region of the target object in the adjacent images is associated through the object candidate regions corresponding to the adjacent images in the video stream, so as to obtain the path information of the target object in the image group. Optionally, the first image and the second image are adjacent images.

[0246] For example, the object recognition model identifies the position rectangular frames Box_a and Box_b of the target object in the pictures A and B and obtains the trajectory Box_ab from the initial two pictures A and B, and then stores the Box_a and Box_b in the candidate frame pool corresponding to the pictures B and C, and executes the above object recognition process again. In this way, the rectangular frame trajectory Box_ab between the pictures A and B can be connected with the rectangular frame trajectory Box_bc between the pictures B and C, and the subsequent process is sequentially similar, so as to obtain the complete motion trajectory of the target object in the multiple pictures.

[0247] In summary, the technical scheme provided by the embodiments of the present disclosure can determine the object recognition result corresponding to the object candidate region of the at least two continuous images in the video stream, determine whether the object candidate region of the at least two images includes the same target object through the object recognition result, if the determination result is yes, the same object in the continuous images is determined through the object candidate region across the images, the complexity and time consumption of object recognition association between different images are reduced, and the accuracy of object recognition association is improved. After it is determined that the object candidate region of the continuous images includes the same target object, the position information of the target object in the at least two images is determined through the object recognition result, and then the path information corresponding to the target object in the video stream is determined, which improves the efficiency and accuracy of object path determination as a whole.

[0248] The technical scheme provided by the embodiments of the present disclosure will be described below in combination with a specific example. In one example, as shown in Figure 7 Fig. 1 shows a flowchart of a process of determining path information based on an object recognition model. Figure 7In the specific implementation, two consecutive query frames, i.e., picture 701 and picture 702, in a video stream are input into a backbone network 703 of an object recognition model. The picture 701 is the t-th frame in the video stream, and the picture 702 is the t+1-th frame in the video stream. Image feature extraction processing is performed on the picture 701 and the picture 702 by the backbone network 703 to obtain a feature map 704 corresponding to the picture 701 and a feature map 705 corresponding to the picture 702. The feature map 704 can be denoted as F_t, and the feature map 705 can be denoted as F_t+1. The feature map 704 and the feature map 705 output by the backbone network 703 are input into a region proposal network (RPN) 706 of the object recognition model. The region proposal network (RPN) 706 performs a region proposal operation based on the feature map 704 and the feature map 705 to output a predicted region proposal 707 corresponding to the picture 701 and a predicted region proposal 708 corresponding to the picture 702, and to form a region proposal pool 709. The region proposal 707 corresponding to the picture 701 can be denoted as P_t, the region proposal 708 corresponding to the picture 702 can be denoted as P_t+1, and the region proposal pool can be denoted as P, i.e., P=(P_t, P_t+1). For the picture 701, the feature map 704 and the feature map 705 are connected in parallel to obtain a feature fusion map 710 corresponding to the picture 701, which can be denoted as cat(F_t, F_t+1). For the picture 702, the feature map 705 and the feature map 704 are connected in parallel to obtain a feature fusion map 711 corresponding to the picture 702, which can be denoted as cat(F_t+1, F_t).

[0249] After the feature parallel connection, a region proposal 712 is selected from the region proposal pool 709, and deformable feature extraction operations (Deform Align) are performed on the feature fusion map 710 and the feature fusion map 711 based on the region proposal 712 to extract an object feature 713 of a picture content corresponding to the region proposal 712 in the picture 701 and an object feature 714 of a picture content corresponding to the region proposal 712 in the picture 702. The region proposal 712 is any region proposal in the region proposal pool 709.

[0250] Finally, object classification, object discrimination, and region regression processing are performed on the object feature 713 and the object feature 714 by a detection branch network (RCNN head) to obtain an object recognition result 715 corresponding to the picture 701 and an object recognition result 716 corresponding to the picture 702.

[0251] If the object class information in the object recognition result 715 indicates that the picture 701 and the picture 702 are of the same object class corresponding to the candidate box 712, and the object discrimination score in the object recognition result 715 is higher than the discrimination score threshold, the position box 717 in the object recognition result 715 is output as the position information of the target object 718 in the picture 701.

[0252] If the object class information in the object recognition result 716 indicates that the picture 702 and the picture 701 are of the same object class corresponding to the candidate box 712, and the object discrimination score in the object recognition result 716 is higher than the discrimination score threshold, the position box 719 in the object recognition result 716 is output as the position information of the target object 718 in the picture 702.

[0253] The above is the object recognition process of the object recognition model on the two continuous images. For the video frames in the entire video, two adjacent frames can be processed as an image group, for example, the picture 700 and the picture 701 are input into the object recognition model as an image group for processing, and the picture 701 and the picture 702 are input into the object recognition model as an image group for processing, and then the position boxes of the same object in the continuous multiple frames can be obtained. The position boxes of the same object in the continuous multiple frames are associated, and the path information can be obtained. The picture 700 is the t-1 frame in the video stream.

[0254] For example, the object recognition model can obtain the position box 720 of the target object 718 in the picture 700, the position box 717 of the target object 718 in the picture 701, and the position box 719 of the target object 718 in the picture 702. By connecting the position box 720, the position box 717, and the position box 719, the path information of the target object 718 between the picture 701, the picture 702, and the picture 703 can be obtained.

[0255] As can be seen from the above example, the technical scheme provided by the embodiments of the present disclosure can connect the objects of adjacent frames by introducing a reference rectangular box, and identify whether the objects in the reference rectangular box of the two adjacent frames belong to the same object, so as to establish the association relationship of the same object in the adjacent frames through the reference rectangular box, and then determine the path information of multiple objects in the entire video stream. The whole process is simple and efficient, can be trained end to end, and improves the accuracy of the path information and reduces the time consumption of determining the path.

[0256] Figure 8 is a block diagram of an object path information determination apparatus according to an example embodiment. Referring to Figure 8 The apparatus includes an image group acquisition module 810, an object recognition module 820, a position determination module 830, and a path determination module 840.

[0257] The image group obtaining module 810 is configured to perform obtaining at least one image group corresponding to the video stream, and any image group comprises at least two continuous image frames in the video stream.

[0258] The object recognition module 820 is configured to perform determining an object candidate region corresponding to the at least two image frames and an object recognition result corresponding to the object candidate region, and the object recognition result represents object information of an object contained in the object candidate region.

[0259] The position determining module 830 is configured to perform, if the object recognition result indicates that the object candidate region corresponding to the at least two image frames comprises a same target object, determining position information of the target object in the at least two image frames.

[0260] The path determining module 840 is configured to perform determining path information corresponding to the target object based on the position information, and the path information represents a motion trajectory of the target object in the video stream.

[0261] In an example embodiment, the object recognition module 820 comprises:

[0262] The image feature extraction submodule is configured to perform feature extraction processing on the at least two image frames to obtain image features corresponding to the at least two image frames.

[0263] The candidate region determination submodule is configured to perform a candidate box generation operation according to the image features to obtain the object candidate region.

[0264] The region set determination submodule is configured to obtain an object candidate region set based on the object candidate region.

[0265] The object recognition submodule is configured to perform object recognition processing on the at least two image frames based on a target object candidate region to obtain the object recognition result, and the target object candidate region is an object candidate region to be processed in the object candidate region set.

[0266] In an example embodiment, the at least two image frames comprise a first image and a second image, the first image and the second image are adjacent images in the at least two image frames, and the image features comprise a first feature map corresponding to the first image and a second feature map corresponding to the second image.

[0267] The object recognition submodule comprises:

[0268] The feature map fusion unit is configured to perform fusion processing on the first feature map and the second feature map to obtain a fused feature map.

[0269] The object feature extraction unit is configured to perform an object feature extraction operation on the fusion feature map based on the target object candidate region to obtain object features, the object features representing object information associated with the target object candidate region.

[0270] The object feature detection unit is configured to perform an object detection operation according to the object features to obtain the object recognition result.

[0271] In an example embodiment, the object feature detection unit includes:

[0272] The image determination subunit is configured to determine a target image corresponding to the fusion feature map and a verification image, a fusion order of a feature map corresponding to the target image being before a fusion order of a feature map corresponding to the verification image.

[0273] The object classification subunit is configured to perform an object classification process based on the object features to obtain first object category information corresponding to a target object candidate region in the target image and second object category information corresponding to a target object candidate region in the verification image.

[0274] The object discrimination subunit is configured to perform an object discrimination process based on the object features to obtain an object discrimination score, the object discrimination score representing a degree of homology between an object corresponding to the first object category information and an object corresponding to the second object category information.

[0275] The region regression subunit is configured to perform a region regression process on the target object candidate region based on the object features to obtain an object location region corresponding to the target image.

[0276] The object recognition result includes the first object category information, the second object category information, the object discrimination score, and the object location region.

[0277] In an example embodiment, the location determination module 830 includes:

[0278] The target object determination unit is configured to determine that the target object candidate region corresponding to the target image and the target object candidate region corresponding to the verification image include the same target object if the first object category information and the second object category information are the same and the object discrimination score is higher than a discrimination score threshold.

[0279] The location region determination unit is configured to determine the object location region as location information of the target object in the target image.

[0280] In an example embodiment, the fusion feature map comprises a first fusion feature map corresponding to the first image and a second fusion feature map corresponding to the second image, the target image corresponding to the first fusion feature map is the first image, the verification image corresponding to the first fusion feature map is the second image, the target image corresponding to the second fusion feature map is the second image, and the verification image corresponding to the second fusion feature map is the first image.

[0281] In an example embodiment, the object feature extraction unit comprises:

[0282] The extraction region determination subunit is configured to perform determination of the feature extraction region corresponding to the object candidate region.

[0283] The extraction region transformation subunit is configured to perform a transformation operation on the feature extraction region to obtain a transformed feature extraction region.

[0284] The object feature extraction subunit is configured to perform an object feature extraction operation on the fusion feature map based on the transformed feature extraction region to obtain the object feature.

[0285] In an example embodiment, the device 800 further comprises:

[0286] The image group determination module is configured to perform determination of a previous image group corresponding to an image group corresponding to the at least two frames of images.

[0287] The position region acquisition module is configured to perform acquisition of a position region corresponding to an object in the previous image group.

[0288] The region set determination sub-module is further configured to obtain the object candidate region set based on the object candidate region and the position region.

[0289] In an example embodiment, the object recognition module is further configured to input the at least two frames of images into an object recognition model and output the object candidate region and the object recognition result.

[0290] The object recognition model is a machine learning model trained based on a sample video and label information, the sample video comprises at least one sample image group, any sample image group comprises at least two sample images in the sample video in sequence, and the label information comprises an object position annotation region, object identification information, and object category annotation information corresponding to the object identification information corresponding to the at least two sample images.

[0291] In an example embodiment, the device 800 further comprises a model training device, and the model training device comprises:

[0292] The sample object recognition module is configured to input the at least two sample images into a to-be-trained object recognition model to obtain sample object candidate regions, sample object category information, sample object discrimination scores, and sample object position regions corresponding to the at least two sample images.

[0293] The coverage determination module is configured to determine a region coverage between the sample object candidate regions and the object position annotation regions.

[0294] The candidate region classification module is configured to perform classification processing on the sample object candidate regions based on the region coverage to obtain positive sample object candidate regions and negative sample object candidate regions.

[0295] The first loss information determination module is configured to determine first loss information based on the sample object position regions and the object position annotation regions, the first loss information representing a difference degree between the sample object position regions and the object position annotation regions.

[0296] The second loss information determination module is configured to determine second loss information based on the sample object category information and the object category annotation information, the second loss information representing a difference degree between the sample object category information and the object category annotation information.

[0297] The third loss information determination module is configured to determine third loss information based on the sample object discrimination scores and the object identification information, the third loss information representing a correctness degree of the sample object discrimination scores.

[0298] The model parameter update module is configured to perform model parameter update processing according to the positive sample object candidate regions, the negative sample object candidate regions, the first loss information, the second loss information, and the third loss information to obtain the object recognition model.

[0299] In an example embodiment, the at least two sample images include a first sample image and a second sample image, the first sample image and the second sample image being adjacent images, and the object position annotation regions include a first annotation region corresponding to the first sample image and a second annotation region corresponding to the second sample image.

[0300] The coverage determination module is specifically configured to determine a first coverage between the sample object candidate regions and the first annotation region, and a second coverage between the sample object candidate regions and the second annotation region.

[0301] The candidate region classification module includes:

[0302] The positive sample determination unit is configured to determine the sample object candidate region as the positive sample object candidate region if the first coverage is greater than a coverage threshold and the second coverage is greater than the coverage threshold.

[0303] The negative sample determination unit is configured to determine the sample object candidate region as the negative sample object candidate region if the first coverage is less than or equal to the coverage threshold or the second coverage is less than or equal to the coverage threshold.

[0304] In summary, the embodiments of the present disclosure provide a technical solution that determines the object candidate region corresponding to at least two consecutive images in a video stream, determines the object recognition result corresponding to the object candidate region, determines whether the object candidate region corresponding to the at least two images includes the same target object through the object recognition result, determines the same object in the consecutive images through the object candidate region across the images if the determination result is yes, reduces the complexity and time consumption of object recognition association between different images, and improves the accuracy of object recognition association. After determining that the object candidate region corresponding to the consecutive images includes the same target object, the position information of the target object in the at least two images is determined through the object recognition result, and then the path information of the target object in the video stream is determined, thereby improving the efficiency and accuracy of object path determination as a whole.

[0305] As to the apparatus in the above embodiments, the specific manners in which various modules perform operations have been described in detail in the embodiments of the method, and will not be described here in detail.

[0306] Figure 9 is a block diagram of an electronic device for determining object path information according to an example embodiment. The electronic device can be a terminal, and its internal structure can be as shown in Figure 9 The electronic device includes a processor, a memory, a network interface, a display screen, and an input device connected through a system bus. The processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The network interface of the electronic device is used to communicate with external terminals through network connection. The computer program is executed by the processor to implement a method for determining object path information. The display screen of the electronic device can be a liquid crystal display screen or an electronic ink display screen. The input device of the electronic device can be a touch layer overlaid on the display screen, or a key, trackball, or touchpad provided on the shell of the electronic device. It can also be an external keyboard, touchpad, or mouse, etc.

[0307] Those skilled in the art can understand that, Figure 9 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present disclosure, and does not constitute a limitation on the electronic device to which the scheme of the present disclosure is applied. The specific electronic device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0308] In an example embodiment, an electronic device is also provided, comprising: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the method for determining object path information as in the embodiments of the present disclosure.

[0309] In an example embodiment, a computer readable storage medium is also provided, when the instructions in the storage medium are executed by the processor of an electronic device, the electronic device is enabled to perform the method for determining object path information as in the embodiments of the present disclosure.

[0310] In an example embodiment, a computer program product containing instructions is also provided, when it is run on a computer, the computer is enabled to perform the method for determining object path information as in the embodiments of the present disclosure.

[0311] Persons of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. Any reference to memory, storage, database or other medium used in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0312] Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the features disclosed herein. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.

[0313] It should be understood that the present disclosure is not limited to the precise structures as herein described and illustrated in the drawings, and that various modifications and changes can be made without departing from its scope. The scope of the present disclosure is limited only by the claims that follow.

Claims

1. A method for determining object path information, characterized in that, include: Obtain at least one image group corresponding to the video stream, wherein any image group includes at least two consecutive frames of images in the video stream; The at least two frames of images include an adjacent first image and a second image; The process involves determining object candidate regions corresponding to at least two frames of images and object recognition results corresponding to those object candidate regions. The object recognition results characterize object information contained within the object candidate regions. These results include first object category information, second object category information, object identification score, and object location region. The determination is achieved by: identifying a target image and a verification image corresponding to the fused feature map; the fusion order of the feature maps corresponding to the target image precedes that of the feature map corresponding to the verification image; and the fused feature map is the image resulting from the fusion of the first image and the second image. Object classification is performed based on object features to obtain first object category information corresponding to the target object candidate region in the target image and second object category information corresponding to the target object candidate region in the verification image; the object features are obtained by performing object feature extraction operations on the fused feature map based on the target object candidate region. Based on the object features, object identification processing is performed to obtain an object identification score. The object identification score represents the degree of homology between the object corresponding to the first object category information and the object corresponding to the second object category information. Based on the object features, region regression processing is performed on the candidate regions of the target object to obtain the object location regions corresponding to the target image; If the object recognition result indicates that the same target object is included in the object candidate region corresponding to the at least two frames of images, then the position information of the target object in the at least two frames of images is determined; Based on the location information, the path information corresponding to the target object is determined, and the path information represents the motion trajectory of the target object in the video stream.

2. The method according to claim 1, characterized in that, Determining the object candidate regions corresponding to the at least two frames of images and the object recognition results corresponding to the object candidate regions includes: Feature extraction processing is performed on the at least two frames of images to obtain the image features corresponding to the at least two frames of images; Based on the image features, a candidate box generation operation is performed to obtain the candidate region of the object; Based on the object candidate regions, a set of object candidate regions is obtained; Based on the target object candidate region, object recognition processing is performed on the at least two frames of images to obtain the object recognition result, wherein the target object candidate region is the object candidate region to be processed in the set of object candidate regions.

3. The method according to claim 2, characterized in that, The at least two frames of images include a first image and a second image, wherein the first image and the second image are adjacent images in the at least two frames of images, and the image features include a first feature map corresponding to the first image and a second feature map corresponding to the second image; The process of performing object recognition processing on at least two frames of images based on the target object candidate region to obtain the object recognition result includes: The first feature map and the second feature map are fused to obtain a fused feature map. Based on the target object candidate region, an object feature extraction operation is performed on the fused feature map to obtain object features, which represent the object information associated with the target object candidate region; An object detection operation is performed based on the object characteristics to obtain the object recognition result.

4. The method according to claim 1, characterized in that, If the object recognition result indicates that the object candidate regions corresponding to the at least two frames of images include the same target object, then determining the position information of the target object in the at least two frames of images includes: If the first object category information is the same as the second object category information, and the object identification score is higher than the identification score threshold, then it is determined that the target object candidate region corresponding to the target image and the target object candidate region corresponding to the verification image include the same target object. The object location region is determined as the location information of the target object in the target image.

5. The method according to claim 4, characterized in that, The fusion feature map includes a first fusion feature map corresponding to the first image and a second fusion feature map corresponding to the second image. The target image corresponding to the first fusion feature map is the first image, the verification image corresponding to the first fusion feature map is the second image, the target image corresponding to the second fusion feature map is the second image, and the verification image corresponding to the second fusion feature map is the first image.

6. The method according to claim 3, characterized in that, The step of performing object feature extraction on the fused feature map based on the object candidate region to obtain object features includes: Determine the feature extraction region corresponding to the candidate region of the object; The feature extraction region is transformed to obtain the transformed feature extraction region; Based on the transformed feature extraction region, object feature extraction is performed on the fused feature map to obtain the object features.

7. The method according to claim 2, characterized in that, The method further includes: Determine the previous image group corresponding to the image group corresponding to the at least two frames; Obtain the location region corresponding to the object in the previous image group; The process of obtaining a set of object candidate regions based on the object candidate regions includes: Based on the object candidate region and the location region, the object candidate region set is obtained.

8. The method according to any one of claims 1 to 7, characterized in that, Determining the object candidate regions corresponding to the at least two frames of images and the object recognition results corresponding to the object candidate regions includes: The at least two frames of images are input into the object recognition model, and the object candidate region and the object recognition result are output. The object recognition model is a machine learning model trained on sample videos and label information. The sample videos include at least one group of sample images, and each group of sample images includes at least two consecutive sample images in the sample videos. The label information includes the object location annotation region corresponding to the at least two sample images, object identification information, and object category annotation information corresponding to the object identification information.

9. The method according to claim 8, characterized in that, The training process of the object recognition model includes: Input the at least two sample images into the object recognition model to be trained to obtain the sample object candidate region, sample object category information, sample object discrimination score and sample object location region corresponding to the at least two sample images; Determine the region coverage between the candidate region of the sample object and the object location annotation region; Based on the region coverage, the candidate regions of the sample objects are classified to obtain positive and negative candidate regions of the sample objects. Based on the sample object location region and the object location annotation region, first loss information is determined, wherein the first loss information characterizes the degree of difference between the sample object location region and the object location annotation region; Based on the sample object category information and the object category labeling information, a second loss information is determined, wherein the second loss information characterizes the degree of difference between the sample object category information and the object category labeling information; Based on the sample object identification score and the object identification information, a third loss information is determined, which characterizes the correctness of the sample object identification score. The model parameters are updated based on the positive sample object candidate region, the negative sample object candidate region, the first loss information, the second loss information, and the third loss information to obtain the object recognition model.

10. The method according to claim 9, characterized in that, The at least two sample images include a first sample image and a second sample image, wherein the first sample image and the second sample image are adjacent images, and the object location annotation region includes a first annotation region corresponding to the first sample image and a second annotation region corresponding to the second sample image. Determining the region coverage between the candidate sample object region and the object location annotation region includes: A first coverage rate is determined between the candidate region of the sample object and the first labeled region, and a second coverage rate is determined between the candidate region of the sample object and the second labeled region, wherein the region coverage rate includes the first coverage rate and the second coverage rate; The step of classifying the candidate regions of sample objects based on the region coverage to obtain positive and negative candidate regions of sample objects includes: If the first coverage rate is greater than the coverage rate threshold and the second coverage rate is greater than the coverage rate threshold, then the candidate region of the sample object is determined as the candidate region of the positive sample object; If the first coverage rate is less than or equal to the coverage threshold or the second coverage rate is less than or equal to the coverage threshold, then the candidate region of the sample object is determined as the candidate region of the negative sample object.

11. A device for determining object path information, characterized in that, include: The image group acquisition module is configured to acquire at least one image group corresponding to the video stream, wherein any image group includes at least two consecutive frames of images in the video stream; the at least two frames of images include an adjacent first image and a second image; The object recognition module is configured to determine object candidate regions corresponding to the at least two frames of images and object recognition results corresponding to the object candidate regions. The object recognition results characterize object information of the objects contained in the object candidate regions. The object recognition results include first object category information, second object category information, object identification score, and object location region. The image determination subunit is configured to determine the target image and the verification image corresponding to the fused feature map, wherein the fusion order of the feature maps corresponding to the target image is before that of the feature map corresponding to the verification image; the fused feature map is the image obtained by fusing the first image and the second image. The object classification subunit is configured to perform object classification processing based on object features to obtain first object category information corresponding to the target object candidate region in the target image and second object category information corresponding to the target object candidate region in the verification image; the object features are obtained by performing object feature extraction operations on the fused feature map based on the target object candidate region. The object identification subunit is configured to perform object identification processing based on the object features to obtain an object identification score, wherein the object identification score characterizes the degree of homology between the object corresponding to the first object category information and the object corresponding to the second object category information. The region regression subunit is configured to perform region regression processing on the candidate regions of the target object based on the object features, so as to obtain the object location region corresponding to the target image. The location determination module is configured to determine the location information of the target object in the at least two frames of images if the object recognition result indicates that the same target object is included in the object candidate regions corresponding to the at least two frames of images. The path determination module is configured to determine path information corresponding to the target object based on the location information, wherein the path information represents the motion trajectory of the target object in the video stream.

12. The apparatus according to claim 11, characterized in that, The object recognition module includes: The image feature extraction submodule is configured to perform feature extraction processing on the at least two frames of images to obtain image features corresponding to the at least two frames of images; The candidate region determination submodule is configured to perform a candidate box generation operation based on the image features to obtain the object candidate region; The region set determination submodule is configured to perform operations to obtain a set of candidate regions for the object based on the candidate regions of the object; The object recognition submodule is configured to perform object recognition processing on the at least two frames of images based on the target object candidate region to obtain the object recognition result, wherein the target object candidate region is the object candidate region to be processed in the set of object candidate regions.

13. The apparatus according to claim 12, characterized in that, The at least two frames of images include a first image and a second image, wherein the first image and the second image are adjacent images in the at least two frames of images, and the image features include a first feature map corresponding to the first image and a second feature map corresponding to the second image; The object recognition submodule includes: The feature map fusion unit is configured to perform a fusion process on the first feature map and the second feature map to obtain a fused feature map; The object feature extraction unit is configured to perform an object feature extraction operation on the fused feature map based on the target object candidate region to obtain object features, wherein the object features characterize the object information associated with the target object candidate region; The object feature detection unit is configured to perform an object detection operation based on the object features to obtain the object recognition result.

14. The apparatus according to claim 11, characterized in that, The location determination module includes: The target object determination unit is configured to perform the following: if the first object category information is the same as the second object category information and the object identification score is higher than the identification score threshold, then determine that the target object candidate region corresponding to the target image and the target object candidate region corresponding to the verification image include the same target object. The location region determination unit is configured to determine the object location region as the location information of the target object in the target image.

15. The apparatus according to claim 14, characterized in that, The fusion feature map includes a first fusion feature map corresponding to the first image and a second fusion feature map corresponding to the second image. The target image corresponding to the first fusion feature map is the first image, the verification image corresponding to the first fusion feature map is the second image, the target image corresponding to the second fusion feature map is the second image, and the verification image corresponding to the second fusion feature map is the first image.

16. The apparatus according to claim 13, characterized in that, The object feature extraction unit includes: The extraction region determination subunit is configured to perform the determination of the feature extraction region corresponding to the candidate region of the object; The feature extraction region transformation subunit is configured to perform a transformation operation on the feature extraction region to obtain the transformed feature extraction region. The object feature extraction subunit is configured to perform object feature extraction operations on the fused feature map based on the transformed feature extraction region to obtain the object features.

17. The apparatus according to claim 12, characterized in that, The device further includes: The image group determination module is configured to determine the previous image group corresponding to the image group corresponding to the at least two frames of images; The location region acquisition module is configured to acquire the location region corresponding to the object in the previous image group; The region set determination submodule is further configured to perform an operation to obtain the object candidate region set based on the object candidate region and the location region.

18. The apparatus according to any one of claims 11 to 17, characterized in that, The object recognition module is further configured to input the at least two frames of images into an object recognition model and output the object candidate region and the object recognition result; The object recognition model is a machine learning model trained on sample videos and label information. The sample videos include at least one group of sample images, and each group of sample images includes at least two consecutive sample images in the sample videos. The label information includes the object location annotation region corresponding to the at least two sample images, object identification information, and object category annotation information corresponding to the object identification information.

19. The apparatus according to claim 18, characterized in that, The apparatus further includes a model training device, the model training device comprising: The sample object recognition module is configured to input the at least two sample images into the object recognition model to be trained, and obtain the sample object candidate region, sample object category information, sample object identification score and sample object location region corresponding to the at least two sample images; The coverage determination module is configured to determine the region coverage between the candidate region of the sample object and the object location annotation region; The candidate region classification module is configured to perform classification processing on the candidate regions of the sample objects based on the region coverage, to obtain positive sample object candidate regions and negative sample object candidate regions. The first loss information determination module is configured to determine first loss information based on the sample object location region and the object location annotation region, wherein the first loss information characterizes the degree of difference between the sample object location region and the object location annotation region; The second loss information determination module is configured to determine second loss information based on the sample object category information and the object category labeling information, wherein the second loss information characterizes the degree of difference between the sample object category information and the object category labeling information; The third loss information determination module is configured to determine third loss information based on the sample object identification score and the object identification information, wherein the third loss information characterizes the correctness of the sample object identification score; The model parameter update module is configured to perform model parameter update processing based on the positive sample object candidate region, the negative sample object candidate region, the first loss information, the second loss information, and the third loss information to obtain the object recognition model.

20. The apparatus according to claim 19, characterized in that, The at least two sample images include a first sample image and a second sample image, the first sample image and the second sample image are adjacent images, and the object location annotation region includes a first annotation region corresponding to the first sample image and a second annotation region corresponding to the second sample image; The coverage determination module is specifically configured to determine a first coverage rate between the candidate region of the sample object and the first labeled region, and a second coverage rate between the candidate region of the sample object and the second labeled region, wherein the region coverage rate includes the first coverage rate and the second coverage rate; The candidate region classification module includes: The positive sample determination unit is configured to determine the candidate region of the sample object as the candidate region of the positive sample object if the first coverage rate is greater than the coverage rate threshold and the second coverage rate is greater than the coverage threshold. The negative sample determination unit is configured to determine the sample object candidate region as the negative sample object candidate region if the first coverage rate is less than or equal to the coverage threshold or the second coverage rate is less than or equal to the coverage threshold.

21. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the method for determining object path information as described in any one of claims 1 to 10.

22. A computer-readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor of the electronic device, the electronic device is able to perform the method for determining object path information as described in any one of claims 1 to 10.

Images