Target detection method and device, storage medium and electronic equipment

Abstract

The invention discloses a target detection method and device, a storage medium and electronic equipment. The method comprises the following steps: acquiring a first image and a second image obtained by shooting the same scene; wherein the shooting moments and / or shooting viewing angles of the first image and the second image are different; performing feature extraction on the first image to obtain a first feature map, and performing feature extraction on the second image to obtain a second feature map; determining a first similarity between a feature of a first pixel point in the first feature map and a feature of a second pixel point in the second feature map; on the basis of the second feature map and the first similarity, updating the features of the first pixel points in the first feature map to obtain a third feature map; on the basis of the first feature map and the first similarity, updating features of second pixel points in the second feature map to obtain a fourth feature map; and determining a detection result of the to-be-detected object based on the third feature map and the fourth feature map. According to the embodiment of the invention, the target detection precision of the image can be well ensured.

Description

Technical Field

[0001] The present disclosure relates to computer vision technology, and in particular to a target detection method, device, storage medium and electronic device. Background Art

[0002] In some scenarios, it is necessary to perform object detection on an image, for example, to find objects of interest in an image, determine the position and size of these objects, etc. How to ensure the accuracy of object detection in an image is a problem worthy of attention for those skilled in the art. Summary of the invention

[0003] In order to solve the above technical problems, the present disclosure provides a target detection method, device, storage medium and electronic device.

[0004] According to one aspect of an embodiment of the present disclosure, there is provided a target detection method, including:

[0005] Acquire a first image and a second image obtained by photographing the same scene; wherein the first image and the second image are photographed at different times and / or at different viewing angles;

[0006] Performing feature extraction on the first image to obtain a first feature map, and performing feature extraction on the second image to obtain a second feature map;

[0007] Determine a first similarity between a feature of a first pixel in the first feature map and a feature of a second pixel in the second feature map;

[0008] Based on the second feature map and the first similarity, updating the feature of the first pixel in the first feature map to obtain a third feature map;

[0009] Based on the first feature map and the first similarity, updating the feature of the second pixel in the second feature map to obtain a fourth feature map;

[0010] Based on the third feature map and the fourth feature map, a detection result of the object to be detected is determined.

[0011] According to another aspect of the embodiments of the present disclosure, there is provided a target detection device, including:

[0012] An acquisition module, used to acquire a first image and a second image obtained by photographing the same scene; wherein the first image and the second image are photographed at different times and / or at different viewing angles;

[0013] a feature extraction module, configured to extract features from the first image obtained by the acquisition module to obtain a first feature map, and to extract features from the second image obtained by the acquisition module to obtain a second feature map;

[0014] A first determination module, used to determine a first similarity between a feature of a first pixel point in the first feature map obtained by the feature extraction module and a feature of a second pixel point in the second feature map obtained by the feature extraction module;

[0015] A first updating module, configured to update the feature of the first pixel in the first feature map obtained by the feature extraction module based on the second feature map obtained by the feature extraction module and the first similarity determined by the first determination module, so as to obtain a third feature map;

[0016] A second updating module, configured to update the feature of the second pixel in the second feature map obtained by the feature extraction module based on the first feature map obtained by the feature extraction module and the first similarity determined by the first determination module, so as to obtain a fourth feature map;

[0017] The second determination module is used to determine the detection result of the object to be detected based on the third feature map obtained by the first updating module and the fourth feature map obtained by the second updating module.

[0018] According to another aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and the computer program is used to execute the above-mentioned target detection method.

[0019] According to another aspect of the embodiments of the present disclosure, an electronic device is provided, the electronic device comprising:

[0020] processor;

[0021] a memory for storing instructions executable by the processor;

[0022] The processor is used to read the executable instructions from the memory and execute the instructions to implement the above-mentioned target detection method.

[0023] According to another aspect of the embodiments of the present disclosure, a computer program product is provided. When instructions in the computer program product are executed by a processor, the above-mentioned target detection method is performed.

[0024] Based on the target detection method, device, storage medium, electronic device and program product provided by the above embodiments of the present disclosure, the first image and the second image are images obtained by shooting the same scene, and the shooting time and / or shooting angle of the first image and the second image are different. Then, the information carried by each of the first image and the second image has both the same part and different parts. By extracting features from the first image, a first feature map can be obtained. By extracting features from the second image, a second feature map can be obtained. Since the information carried by each of the first image and the second image has both the same part and different parts, the features included in each of the first feature map and the second feature map are different. In addition, the first similarity between the feature of the first pixel in the first feature map and the feature of the second pixel in the second feature map can be determined. Based on the second feature map and the first similarity, the feature of the first pixel in the first feature map can be updated, which is equivalent to performing pixel-level feature interaction on the first feature map and the second feature map, thereby obtaining a more accurate and effective third feature map than the first feature map, for example, the features in the third feature map can more accurately and effectively represent the relevant information of the object to be detected. The relevant information of the object to be detected may include, for example, but is not limited to the position and size of the object to be detected. Based on the first feature map and the first similarity, the feature of the second pixel in the second feature map can be updated, which is equivalent to performing pixel-level feature interaction on the first feature map and the second feature map, thereby obtaining a fourth feature map that is more accurate and effective than the second feature map. For example, the features in the fourth feature map can more accurately and effectively represent the relevant information of the object to be detected. Then, based on the third feature map and the fourth feature map, the detection result of the object to be detected is determined, which can better guarantee the accuracy and reliability of the detection result of the object to be detected. Therefore, by adopting the embodiments of the present disclosure, the target detection accuracy of the image can be better guaranteed. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a flowchart of a target detection method provided by some exemplary embodiments of the present disclosure.

[0026] Figure 2 It is a flowchart of a method for updating a feature of a first pixel in a first feature map based on a second feature map and a first similarity provided by some exemplary embodiments of the present disclosure.

[0027] Figure 3 It is a schematic diagram of some exemplary embodiments of the present disclosure for realizing target detection.

[0028] Figure 4 It is a flowchart of a method for updating the features of a first pixel point based on point information of a target second pixel point provided by some exemplary embodiments of the present disclosure.

[0029] Figure 5 It is a flowchart of a method for selecting a target second pixel point that satisfies a preset feature similarity condition with a first pixel point from each second pixel point in a second feature map based on a first similarity, provided by some exemplary embodiments of the present disclosure.

[0030] Figure 6 It is a flowchart of a method for determining a detection result of an object to be detected based on a third feature map and a fourth feature map provided by some exemplary embodiments of the present disclosure.

[0031] Figure 7 is a schematic diagram of the attention mechanism in some exemplary embodiments of the present disclosure.

[0032] Figure 8 It is a flowchart of a method for generating a detection result of an object to be detected based on a second query vector, a third query vector, a third feature map, a fourth feature map, a second information interaction network and a prediction network, provided by some exemplary embodiments of the present disclosure.

[0033] Fig. 9 It is a flowchart of a method for updating a third query vector to obtain a fourth query vector based on a third feature map and a second information interaction network provided by some exemplary embodiments of the present disclosure.

[0034] Fig.10 It is a flowchart of a method for updating a first query vector of an object to be detected obtained through an initialization operation to obtain a second query vector based on a third feature map and a first information interaction network, provided by some exemplary embodiments of the present disclosure.

[0035] Fig.11 It is a schematic diagram of some other exemplary embodiments of the present disclosure for realizing target detection.

[0036] Fig.12 It is a schematic diagram of the structure of a target detection device provided by some exemplary embodiments of the present disclosure.

[0037] Fig.13 is a schematic diagram of the structure of the first update module in some exemplary embodiments of the present disclosure.

[0038] Figure 14-1 is a schematic diagram of the structure of the first updating submodule in some exemplary embodiments of the present disclosure.

[0039] Figure 14-2 It is a structural schematic diagram of selected submodules in some exemplary embodiments of the present disclosure.

[0040] Fig.15 is a schematic diagram of the structure of the second determination module in some exemplary embodiments of the present disclosure.

[0041] Fig.16It is a schematic diagram of the structure of the generation submodule in some exemplary embodiments of the present disclosure.

[0042] Fig.17 is a schematic structural diagram of a second updating unit in some exemplary embodiments of the present disclosure.

[0043] Fig.18 is a schematic diagram of the structure of the second updating submodule in some exemplary embodiments of the present disclosure.

[0044] Fig.19 It is a schematic diagram of the structure of an electronic device provided by some exemplary embodiments of the present disclosure. DETAILED DESCRIPTION

[0045] To explain the present disclosure, example embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. It should be understood that the present disclosure is not limited to the example embodiments.

[0046] It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.

[0047] Application Overview

[0048] Image object detection technology can be applied to many fields. For example, image object detection technology can be applied to the industrial field to assist in defect detection. For another example, image object detection technology can be applied to the medical field to assist in lesion detection. For another example, image object detection technology can be applied to the driving field to assist in environmental perception.

[0049] How to ensure the accuracy of target detection in images is a problem worthy of attention for those skilled in the art.

[0050] Exemplary Systems

[0051] In the embodiments of the present disclosure, the same scene can be photographed to obtain a first image and a second image with different shooting times and / or shooting angles, so that the information carried by the first image and the second image has both the same part and different parts. By extracting features from the first image, a first feature map can be obtained. By extracting features from the second image, a second feature map can be obtained. In addition, the similarity between the features of at least some pixels in the first feature map and the features of at least some pixels in the second feature map can be determined. Based on the determined similarity, the first feature map and the second feature map can be subjected to pixel-level feature interaction to obtain a third feature map that is more accurate and effective than the first feature map, and a fourth feature map that is more accurate and effective than the second feature map. Based on the third feature map and the fourth feature map, the detection result of the object to be detected can be determined, and the detection result of the result to be detected can be used as the target detection result, and the accuracy and reliability of the target detection result can be better guaranteed. Therefore, by adopting the embodiments of the present disclosure, the target detection accuracy of the image can be better guaranteed.

[0052] Exemplary Methods

[0053] Figure 1 It is a flowchart of a target detection method provided by some exemplary embodiments of the present disclosure. Figure 1 The method shown may include step 110 , step 120 , step 130 , step 140 , step 150 , and step 160 .

[0054] Step 110, acquiring a first image and a second image obtained by photographing the same scene; wherein the first image and the second image are photographed at different times and / or at different viewing angles.

[0055] Optionally, the first image and the second image may be images taken successively by the same camera installed in the vehicle when the vehicle is stationary, and accordingly, the shooting time of the first image and the shooting time of the second image may be different, and the shooting angle of the first image and the shooting angle of the second image may be the same. Alternatively, the first image and the second image may be images taken successively by the same camera installed in the vehicle when the vehicle is turning, and accordingly, the shooting time of the first image and the shooting time of the second image may be different, and the shooting angle of the first image and the shooting angle of the second image may also be different. Alternatively, the first image and the second image may be images taken simultaneously by different cameras installed in the vehicle, and accordingly, the shooting time of the first image and the shooting time of the second image may be the same, and the shooting angle of the first image and the shooting angle of the second image may be different.

[0056] In one example, the vehicle may be provided with six cameras, namely, a first camera disposed in front of the vehicle, a second camera disposed in front of the left side of the vehicle, a third camera disposed in front of the right side of the vehicle, a fourth camera disposed in front of the vehicle, a fifth camera disposed in the rear left side of the vehicle, and a sixth camera disposed in the rear right side of the vehicle. At time t1 and time t2 different from time t1, the first camera may shoot the road scene, the image shot by the first camera at time t1 may be used as the first image, and the image shot by the first camera at time t2 may be used as the second image. Alternatively, at time t1, the first camera and the second camera may both shoot the road scene, the image shot by the first camera at time t1 may be used as the first image, and the image shot by the second camera at time t1 may be used as the second image.

[0057] Optionally, the vehicle on which the camera for capturing the first image and the second image is disposed may also be referred to as the host vehicle hereinafter.

[0058] Step 120: extract features from the first image to obtain a first feature map, and extract features from the second image to obtain a second feature map.

[0059] Optionally, a feature extractor may be used to perform feature extraction on the first image to obtain a first feature map. Similarly, a feature extractor may be used to perform feature extraction on the second image to obtain a second feature map. The feature extractor may be, for example, a feature extraction network in a pre-trained target detection model. The size of the first feature map and the size of the second feature map may be the same or different. For ease of understanding, the following description will take the case where the size of the first feature map and the size of the second feature map are the same as an example.

[0060] Step 130: Determine a first similarity between a feature of a first pixel in the first feature map and a feature of a second pixel in the second feature map.

[0061] Optionally, there may be a total of M pixels in the first feature map, and each of the M pixels may be used as a first pixel, that is, the first feature map may include M first pixels. Similarly, there may be a total of M pixels in the second feature map, and each of the M pixels may be used as a second pixel, that is, the second feature map may include M second pixels. Each of the M first pixels included in the first feature map and each of the M second pixels included in the second feature map may have corresponding features, respectively, and the features of any pixel are used to implicitly represent certain semantic information.

[0062] Optionally, the similarity between the features of at least some of the M first pixels and the features of at least some of the M second pixels can be determined according to a preset similarity algorithm, and each similarity determined can be used as a first similarity, and these first similarities can form a similarity set. The preset similarity algorithm can be, for example, a cosine similarity algorithm.

[0063] In one example, for each of the M first pixel points, the similarity between the feature of the first pixel point and the features of the M second pixel points can be determined to obtain M first similarities corresponding to the first pixel point. In this way, M first similarities corresponding to each of the M first pixel points can be obtained, so that a total of M*M first similarities can be obtained. The M*M first similarities can form a similarity set.

[0064] Step 140: Based on the second feature map and the first similarity, update the feature of the first pixel in the first feature map to obtain a third feature map.

[0065] Optionally, based on the second feature map and the similarity set, the features of the M first pixels included in the first feature map can be updated to update the first feature map to a third feature map. The size of the third feature map can be the same as that of the first feature map.

[0066] Step 150: Based on the first feature map and the first similarity, update the feature of the second pixel in the second feature map to obtain a fourth feature map.

[0067] Optionally, based on the first feature map and the similarity set, the features of the M second pixels included in the second feature map can be updated to update the second feature map to a fourth feature map. The size of the fourth feature map can be the same as the size of the second feature map. Since the size of the first feature map and the size of the second feature map can be the same, the size of the third feature map can be the same as the size of the first feature map, and the size of the third feature map can also be the same as the size of the fourth feature map.

[0068] Step 160: Determine a detection result of the object to be detected based on the third feature map and the fourth feature map.

[0069] Optionally, the objects to be detected may include but are not limited to pedestrians, cyclists, other vehicles except the vehicle itself, etc.

[0070] Optionally, the detection result of the object to be detected determined in step 160 can be used as the target detection result for the first image and the second image. The target detection result may include the detection result of the object to be detected in the first image and the detection result of the object to be detected in the second image. The detection result of the object to be detected in the first image can be determined based on at least the third feature map. The detection result of the object to be detected in the first image can be used to characterize which objects to be detected exist in the first image, and / or the position, size, etc. of these objects to be detected in the first image. The detection result of the object to be detected in the second image can be determined based on at least the fourth feature map. The detection result of the object to be detected in the second image can be used to characterize which objects to be detected exist in the second image, and / or the position, size, etc. of these objects to be detected in the second image. Of course, the composition of the target detection result is not limited to this, and examples will be given later.

[0071] In the embodiment of the present disclosure, the first image and the second image are images obtained by shooting the same scene, and the shooting time and / or shooting angle of the first image and the second image are different. Then, the information carried by each of the first image and the second image has both the same part and different parts. By extracting features from the first image, a first feature map can be obtained. By extracting features from the second image, a second feature map can be obtained. Since the information carried by each of the first image and the second image has both the same part and different parts, the features included in each of the first feature map and the second feature map are different. In addition, the first similarity between the feature of the first pixel in the first feature map and the feature of the second pixel in the second feature map can be determined. Based on the second feature map and the first similarity, the feature of the first pixel in the first feature map can be updated, which is equivalent to performing pixel-level feature interaction on the first feature map and the second feature map, thereby obtaining a more accurate and effective third feature map than the first feature map, for example, the features in the third feature map can more accurately and effectively represent the relevant information of the object to be detected. The relevant information of the object to be detected may include, for example, but is not limited to the position and size of the object to be detected. Based on the first feature map and the first similarity, the feature of the second pixel in the second feature map can be updated, which is equivalent to performing pixel-level feature interaction on the first feature map and the second feature map, thereby obtaining a fourth feature map that is more accurate and effective than the second feature map. For example, the features in the fourth feature map can more accurately and effectively represent the relevant information of the object to be detected. Then, based on the third feature map and the fourth feature map, the detection result of the object to be detected is determined, which can better guarantee the accuracy and reliability of the detection result of the object to be detected. Therefore, by adopting the embodiments of the present disclosure, the target detection accuracy of the image can be better guaranteed.

[0072] Figure 2It is a flowchart of a method for updating a feature of a first pixel in a first feature map based on a second feature map and a first similarity provided by some exemplary embodiments of the present disclosure. Figure 2 The illustrated method may include step 210 and step 220 .

[0073] It should be noted that Figure 2 The method shown can be applied to updating the features of each of the M first pixels included in the first feature map. The following mainly introduces the updating scheme of the features of a single first pixel.

[0074] Step 210 , based on the first similarity, select a target second pixel point that satisfies a preset feature similarity condition with the first pixel point from each second pixel point in the second feature map.

[0075] Optionally, the preset feature similarity condition may include the similarity between features being ranked first (specifically for the case where multiple similarities are ranked from first to last), or the similarity between features being greater than a preset similarity. As an example, the preset similarity may be 80%, 85%, 90%, etc., which are not listed here one by one. It should be noted that the features involved in this paragraph refer to the features of the pixel points, including the semantic information of the feature map.

[0076] Optionally, the second feature map may belong to a multi-scale feature map, then the second feature map may include multiple sub-feature maps of different scales, and the multiple sub-feature maps may be represented as R sub-feature maps; wherein R may be an integer greater than or equal to 2. For example, the R sub-feature maps may be 3 sub-feature maps, such as Figure 3 As shown, the three sub-feature graphs may be sub-feature graph 310, sub-feature graph 320 and sub-feature graph 330 in sequence.

[0077] For a certain sub-feature map among the R sub-feature maps, a target second pixel point that satisfies the preset feature similarity condition with the first pixel point can be selected from each second pixel point in the sub-feature map. For example, the preset feature similarity condition may include the similarity between features being ranked first, or second first, etc. Assuming that there are a total of M1 second pixel points in the sub-feature map, the first similarities between the features of the first pixel point and the respective features of the M1 second pixel points can be obtained from the similarity set to obtain M1 first similarities that correspond one to one with the M1 second pixel points. The M1 first similarities can be arranged in order from large to small, and then the first similarity that ranks first (or the first similarity that ranks second first) can be selected therefrom, and the second pixel point corresponding to the selected first similarity can be used as the target second pixel point in the sub-feature map that satisfies the preset feature similarity condition with the first pixel point. In this way, the target second pixel point that satisfies the preset feature similarity condition with the first pixel point can be obtained. Figure 1A corresponding R target second pixel points.

[0078] The above describes the case where there are multiple target second pixel points. In a specific implementation, the number of target second pixel points may also be only one. For example, the second feature map may be a single-scale feature map, and the first similarities between the features of the first pixel point and the features of the M second pixel points in the second feature map may be obtained from the similarity set to obtain M first similarities corresponding to the M second pixel points. The M first similarities may be arranged in order from large to small, and the first similarity with the highest ranking may be selected from them, and the second pixel point corresponding to the selected first similarity may be used as the only target second pixel point.

[0079] Step 220: Update the feature of the first pixel based on the point information of the target second pixel.

[0080] Optionally, the point information of the second target pixel may include but is not limited to the coordinates, features, etc. of the second target pixel. The coordinates of the second target pixel are used to characterize the pixel position of the second target pixel. The features of the second target pixel include semantic information of the feature map. The number of the second target pixel may be one or more, and updating the features of the first pixel based on the point information of the target pixel may include: updating the features of the first pixel based on the point information of one or more second target pixels.

[0081] In some optional embodiments of the present disclosure, Figure 4 As shown, step 220 may include step 410 and step 420. For ease of understanding, Figure 4 In the illustrated implementation manner, a case where the number of target second pixel points is one is taken as an example for description.

[0082] Step 410, based on the point information of the target second pixel point, determine the coordinates and weight of each second pixel point in the second feature map, among the multiple second pixel points that meet the preset sampling condition.

[0083] Optionally, the preset sampling condition includes pixels distributed within a preset range around the second pixel of the target, and / or pixels with a high feature similarity to the second pixel of the target. Based on the point information of the second pixel of the target, the sampling point determination network in the target detection model is used to determine the coordinates and weights of each second pixel in the second feature map, among the multiple second pixels that meet the preset sampling condition. For example, the sampling point determination network is used to predict the coordinates and weights of each of the multiple sampling points according to the coordinates and features of the reference point; wherein the reference point and the multiple sampling points may be different pixels located in the same image. The coordinates and features of the second pixel of the target can be used as input data of the sampling point determination network, and the sampling point determination network can perform operations accordingly to output the coordinates and weights of each of the multiple sampling points, and each of the multiple sampling points can be used as a second pixel in the second feature map that meets the preset sampling condition. In this way, the sampling point determination network can be used to efficiently and quickly determine the coordinates and weights of each of the multiple second pixels that meet the preset sampling condition.

[0084] In some embodiments, the position of the target second pixel point can be used as the center to determine a target area with a set size, and random sampling or regular sampling of pixels can be performed in the target area to obtain multiple sampling points, each of which can be used as a second pixel point that meets the preset sampling condition in the second feature map. Optionally, the target area can be a rectangular area, and accordingly, the set size can include length and width. Alternatively, the target area can be a circular area, and accordingly, the set size can include radius or diameter. In addition, for each of the multiple sampling points, the weight of the sampling point can be determined based on the distance between the position of the sampling point and the position of the target second pixel point, and the weight of the sampling point is the weight of the corresponding second pixel point that meets the preset sampling condition. Optionally, when determining the weight based on the distance, the weight can be negatively correlated with the distance.

[0085] Step 420: Update the features of the first pixel based on the coordinates and weights of each second pixel that meets the preset sampling condition.

[0086] Optionally, the weights of the respective second pixel points satisfying the preset sampling condition may be used to perform weighted averaging on the respective coordinates of the respective second pixel points satisfying the preset sampling condition to obtain weighted coordinates.

[0087] In one example, the plurality of second pixel points satisfying the preset sampling condition are N second pixel points, the N second pixel points correspond one-to-one to the N coordinates, the N second pixel points also correspond one-to-one to the N weights, and the N coordinates are sequentially represented as P 1 , P 2 , P 3,……,P N , the N weights are represented as K 1 , K 2 , K 3 ,……,K N , the weighted coordinate is expressed as P, then:

[0088] P=(K 1 *P 1 +K 2 *P 2 +K 3 *P 3 +……+K N *P N ) / (K 1 +K 2 +K 3 +……+K N )

[0089] After obtaining the weighted coordinates, the features of the second pixel point having the weighted coordinates in the second feature map may be determined (for ease of description, it may be referred to as the target feature hereinafter), and the features of the first pixel point may be updated based on the target feature. For example, the features of the first pixel point may be updated to the target feature. For another example, a mean operation or a weighted average operation may be performed on the target feature and the features of the first pixel point, and the features of the first pixel point may be updated to the mean operation result or the weighted average operation result.

[0090] In the above implementation, based on the point information of the target second pixel point, the coordinates and weights of each second pixel point in the second feature map and the multiple second pixel points that meet the preset sampling conditions can be determined efficiently and quickly. On this basis, the features of the first pixel point can be updated efficiently and quickly by using the weighted average operation. In this way, the first feature map can be updated to the third feature map through the pixel-level feature interaction between the first feature map and the second feature map. Since the update of the features of the first pixel point uses the coordinates and weights of a specific number of second pixel points, the data referenced by the update of the features of the first pixel point is very rich, which can avoid the adverse effects that may be caused by only referring to individual pixels, and is conducive to ensuring the pixel-level feature interaction effect.

[0091] In some embodiments, the number of target second pixel points is multiple, for example, R, then for each of the R target second pixel points, the above method can be used to obtain the corresponding N coordinates and N weights, and based on the corresponding N coordinates and N weights, the corresponding target feature is obtained. Next, the feature of the first pixel point can be updated based on the R target features that correspond one-to-one to the R target second pixel points. For example, the R target features can be averaged or weighted averaged, and the feature of the first pixel point can be updated to the averaged result or weighted averaged result.

[0092] In the embodiments of the present disclosure, based on the first similarity, a target second pixel point can be reasonably selected from each second pixel point in the second feature map, and the point information of the target second pixel point can provide an effective reference for updating the feature of the first pixel point. On this basis, a third feature map that is more accurate and effective than the first feature map can be effectively obtained based on the pixel-level feature interaction between the first feature map and the second feature map. Using the third feature map as the basis for determining the detection result of the object to be detected can better ensure the accuracy and reliability of the detection result of the object to be detected.

[0093] Figure 5 It is a flowchart of a method for selecting a target second pixel point that satisfies a preset feature similarity condition with a first pixel point from each second pixel point in a second feature map based on a first similarity, provided by some exemplary embodiments of the present disclosure. Figure 5 The method shown may include step 510 , step 520 , step 530 , and step 540 .

[0094] Optionally, Figure 5 In the method shown, the second feature map may include a first scale sub-feature map and a second scale sub-feature map. Figure 3 As shown, the second feature map may include sub-feature maps of three scales, namely, sub-feature map 310, sub-feature map 320 and sub-feature map 330. Then, one of the sub-feature map 310, sub-feature map 320 and sub-feature map 330 can be used as a first-scale sub-feature map, and the other two of the sub-feature map 310, sub-feature map 320 and sub-feature map 330 can be used as a second-scale sub-feature map.

[0095] Step 510: Based on the first similarity, select a target second pixel point that satisfies a preset feature similarity condition with the first pixel point from each second pixel point in the first scale sub-feature map.

[0096] Optionally, the preset feature similarity condition can refer to the relevant introduction to step 210 above, which will not be repeated here.

[0097] Optionally, there may be a total of M1 second pixel points in the first scale sub-feature map, and the first similarities between the features of the first pixel point and the respective features of the M1 second pixel points may be obtained from the similarity set to obtain M1 first similarities. The M1 first similarities may be arranged in order from large to small, and the first similarity with the highest ranking may be selected therefrom, and the second pixel point corresponding to the selected first similarity may be used as the target second pixel point in the first scale sub-feature map that satisfies the preset feature similarity condition with the first pixel point.

[0098] Step 520: Determine a size ratio relationship between the second scale sub-feature map and the first scale sub-feature map.

[0099] Optionally, Figure 3 The sub-feature graph 320 in the figure may be obtained by scaling the sub-feature graph 310, and the sub-feature graph 330 may be obtained by scaling the sub-feature graph 320. Then, for the second-scale sub-feature graph and the first-scale sub-feature graph, a first ratio of the width of the second-scale sub-feature graph to the width of the first-scale sub-feature graph may be determined, and the first ratio may be used to characterize the size ratio relationship between the second-scale sub-feature graph and the first-scale sub-feature graph. Alternatively, a second ratio of the height of the second-scale sub-feature graph to the height of the first-scale sub-feature graph may be determined, and the second ratio may be used to characterize the size ratio relationship between the second-scale sub-feature graph and the first-scale sub-feature graph. Alternatively, a third ratio of the width of the first-scale sub-feature graph to the width of the second-scale sub-feature graph may be determined, and the third ratio may be used to characterize the size ratio relationship between the second-scale sub-feature graph and the first-scale sub-feature graph.

[0100] Step 530 : Based on the size ratio relationship, the coordinates of the second pixel point of the target in the first scale sub-feature map are mapped to the second scale sub-feature map to obtain the mapping coordinates on the second scale sub-feature map.

[0101] Optionally, the coordinates of the target second pixel point in the first scale sub-feature map can be determined, and the determined coordinates can be mapped to the second scale sub-feature map. The obtained mapping coordinates can satisfy: the relative distribution of the position with the mapping coordinates in the second scale sub-feature map is called the first relative distribution, and the relative distribution of the position where the target second pixel point in the first scale sub-feature map is located in the first scale sub-feature map is called the second relative distribution, then the first relative distribution and the second relative distribution are consistent.

[0102] In one example, the coordinates of the target second pixel point in the first scale sub-feature map are expressed as (x1, y1). The first ratio in the above text is used to characterize the size ratio relationship between the second scale sub-feature map and the first scale sub-feature map, and the first ratio is expressed as k1, and the mapping coordinates on the second scale sub-feature map are expressed as (x2, y2), then: x2 = k1*x1, y2 = k1*y1.

[0103] Step 540 : Based on the mapping coordinates, select another target second pixel point that satisfies a preset feature similarity condition with the first pixel point from each second pixel point in the second-scale sub-feature map.

[0104] Optionally, a second pixel point with mapping coordinates can be selected from the second scale sub-feature map, and the selected second pixel point can be used as another target second pixel point in the second scale sub-feature map that satisfies a preset feature similarity condition with the first pixel point. The other target second pixel point is a different target second pixel point (located in a different sub-feature map) from the target second pixel point selected in step 510.

[0105] In the embodiment of the present disclosure, when the second feature map belongs to a multi-scale feature map, the target second pixel point can be first determined in a sub-feature map of a certain scale, and then the second pixel points of other targets can be determined in sub-feature maps of other scales by coordinate mapping between different sub-feature maps. In this way, the similarity set does not need to include the first similarity between the first pixel point and each second pixel point in the sub-feature map of the other scale, so the workload of determining the first similarity is reduced, which is conducive to reducing redundant calculations and improving target detection efficiency.

[0106] It should be noted that the above describes in detail the method of updating the features of the first pixel in the first feature map based on the second feature map and the first similarity to obtain the third feature map. The method of updating the features of the second pixel in the second feature map based on the first feature map and the first similarity to obtain the fourth feature map is similar to this, and will not be described in detail in this disclosure.

[0107] Figure 6 It is a flowchart of a method for determining a detection result of an object to be detected based on a third feature map and a fourth feature map provided by some exemplary embodiments of the present disclosure. Figure 6 The illustrated method may include step 610 , step 620 , and step 630 .

[0108] Step 610: Based on the third feature map and the first information interaction network, the first query vector of the object to be detected obtained through the initialization operation is updated to obtain a second query vector.

[0109] Optionally, the first information interaction network may be a network used for information interaction in the target detection model. The first information interaction network may be a network based on an attention mechanism. The attention mechanism may include, but is not limited to, a self-attention mechanism, a cross-attention mechanism, and the like. The self-attention mechanism may, for example, be a multi-scale deformable self-attention mechanism. The cross-attention mechanism may, for example, be a multi-scale deformable cross-attention mechanism. The network based on the attention mechanism may, for example, be a Transformer network, and the Transformer network may adopt an encoding-decoding architecture.

[0110] In general, the principle of the attention mechanism is: calculate the attention distribution on the given information (i.e. determine what information is important and what information is not important); perform weighted average operation based on the attention distribution to obtain the attention calculation result. Figure 7 , specifically: Source includes a series of<Key,Value> Data pairs (also called key-value pairs), for the element Query, calculate the similarity between the Query and each Key, get the weight corresponding to each Key, use these weights to perform weighted summation on each Value to get the final Attention calculation result (ie Attention Value).

[0111] Optionally, a first query vector of the object to be detected can be obtained through an initialization operation. The first query vector can be a vector carrying relevant information of the object to be detected. The initialization operation can be, for example, a random initialization operation. The number of first query vectors can be multiple, for example, V, and the value of V can be determined based on the maximum number of objects to be detected that may appear in a single image. For example, a maximum of 100 objects to be detected may appear in a single image, and V can be 100.

[0112] Optionally, the third feature map and the first query vector can be used as input data of the first information interaction network, and the first information interaction network can perform operations based on the input data to apply the attention mechanism to update the first query vector, thereby obtaining a second query vector that is more accurate and effective than the first query vector. For example, the second query vector can more accurately and effectively reflect the relevant information of the object to be detected. The number of second query vectors can also be V.

[0113] Step 620: Based on the fourth feature graph and the first information interaction network, update the first query vector to obtain a third query vector.

[0114] Optionally, the fourth feature map and the first query vector can be used as input data of the first information interaction network, and the first information interaction network can perform operations based on the input data to apply the attention mechanism to update the first query vector, thereby obtaining a third query vector that is more accurate and effective than the first query vector. For example, the third query vector can more accurately and effectively reflect the relevant information of the object to be detected. The number of third query vectors can also be V.

[0115] Step 630: Generate a detection result of the object to be detected based on the second query vector, the third query vector, the third feature map, the fourth feature map, the second information interaction network and the prediction network.

[0116] In some optional embodiments of the present disclosure, Figure 8 As shown, step 630 may include step 810 , step 820 , step 830 and step 840 .

[0117] Step 810: Based on the third feature map and the second information interaction network, update the third query vector to obtain a fourth query vector.

[0118] Optionally, the second information interaction network may be a network used for information interaction in the target detection model. The second information interaction network may be a network based on an attention mechanism.

[0119] Optionally, the third feature map and the third query vector can be used as input data of the second information interaction network, and the second information interaction network can perform operations based on the input data to apply the attention mechanism to update the third query vector, thereby obtaining a fourth query vector that is more accurate and effective than the third query vector. For example, the fourth query vector can more accurately and effectively reflect the relevant information of the object to be detected. The number of fourth query vectors can also be V.

[0120] Step 820 : Based on the second query vector and the fourth query vector, a detection result of the object to be detected in the first image is generated through a prediction network.

[0121] Optionally, the prediction network may be a network in a target detection model for generating target detection results based on a given query vector.

[0122] Optionally, the V second query vectors and the V fourth query vectors can be used as input data of the prediction network, and the prediction network can decode the input data to obtain the detection result of the object to be detected in the first image. The detection result of the object to be detected in the first image can be represented by the first image with 2V two-dimensional detection boxes.

[0123] Step 830: Based on the fourth feature map and the second information interaction network, update the second query vector to obtain a fifth query vector.

[0124] Optionally, the fourth feature map and the second query vector can be used as input data of the second information interaction network, and the second information interaction network can perform operations based on the input data to apply the attention mechanism to update the second query vector, thereby obtaining a fifth query vector that is more accurate and effective than the second query vector. For example, the fifth query vector can more accurately and effectively reflect the relevant information of the object to be detected. The number of fifth query vectors can also be V.

[0125] Step 840: Generate a detection result of the object to be detected in the second image via a prediction network based on the third query vector and the fifth query vector.

[0126] Optionally, the V third query vectors and the V fifth query vectors can be used as input data of the prediction network, and the prediction network can decode the input data to obtain the detection result of the object to be detected in the second image. The detection result of the object to be detected in the second image can be represented by a second image with 2V two-dimensional detection boxes.

[0127] In the above implementation, based on the second information interaction network, the information interaction between the third feature map and the third query vector can be effectively realized to obtain a fourth query vector that is more accurate and effective than the third query vector. The second query vector and the fourth query vector are used as the basis for generating the detection result of the object to be detected in the first image, which can better ensure the integrity, accuracy and reliability of the generated detection result. In addition, based on the second information interaction network, the information interaction between the fourth feature map and the second query vector can be effectively realized to obtain a fifth query vector that is more accurate and effective than the second query vector. The third query vector and the fifth query vector are used as the basis for generating the detection result of the object to be detected in the second image, which can better ensure the integrity, accuracy and reliability of the generated detection result. In this way, the target detection model can have good detection performance.

[0128] In some embodiments, the detection result of the object to be detected in the first image may be generated via the prediction network based only on the fourth query vector. Similarly, the detection result of the object to be detected in the second image may be generated via the prediction network based only on the fifth query vector.

[0129] In the embodiment of the present disclosure, based on the first information interaction network, the information interaction between the third feature map and the first query vector can be effectively realized to obtain a second query vector that is more accurate and effective than the first query vector. Similarly, based on the first information interaction network, the information interaction between the fourth feature map and the first query vector can be effectively realized to obtain a third query vector that is more accurate and effective than the first query vector. Combined with the third feature map, the fourth feature map, the second information interaction network and the prediction network, the detection result of the object to be detected can be reasonably determined, which is conducive to ensuring the accuracy and reliability of the detection result of the object to be detected.

[0130] Fig. 9 It is a flowchart of a method for updating a third query vector to obtain a fourth query vector based on a third feature map and a second information interaction network provided by some exemplary embodiments of the present disclosure. Fig. 9 The illustrated method may include step 910 and step 920 .

[0131] Step 910: Based on the first self-attention network in the second information interaction network, update the third query vector to obtain a sixth query vector.

[0132] Optionally, the second information interaction network may include a first self-attention network and a first cross-attention network; wherein the first self-attention network may be a network based on a self-attention mechanism, and the first cross-attention network may be a network based on a cross-attention mechanism.

[0133] Optionally, based on V third query vectors, Figure 7 Multiple Sources in<Key,Value> In addition, each of the V third query vectors can be used as a query, so that the corresponding V attention values ​​can be obtained, and each of the V attention values ​​can be used as a sixth query vector, so that there can be V sixth query vectors.

[0134] Step 920: Update the sixth query vector based on the third feature map and the first cross attention network in the second information interaction network to obtain a fourth query vector.

[0135] Optionally, based on the third feature map, Figure 7Multiple Sources in<Key,Value> In addition, each of the V sixth query vectors can be used as a query, so that the corresponding V attention values ​​can be obtained, and each of the V attention values ​​can be used as a fourth query vector, so that there can be V fourth query vectors.

[0136] In the embodiments of the present disclosure, the third query vector can be updated first through the self-attention mechanism. Since each query vector can carry relevant information of an object to be detected, an object to be detected can be used as an instance, which is equivalent to obtaining a sixth query vector that is more accurate and effective than the third query vector through instance-level information interaction. Afterwards, the third feature map and the sixth query vector can be interacted with each other through the cross-attention mechanism to learn useful information from the third feature map for updating the sixth query vector, thereby obtaining a fourth query vector that is more accurate and effective than the sixth query vector. Therefore, through the use of the self-attention mechanism and the cross-attention mechanism, the third query vector can be effectively optimized to obtain a reasonable and reliable fourth query vector.

[0137] In some embodiments, the second information interaction network may also include only the first cross-attention network. Then, the third query vector may be updated directly based on the third feature map and the first cross-attention network to obtain the fourth query vector.

[0138] Fig.10 It is a flowchart of a method for updating a first query vector of an object to be detected obtained through an initialization operation to obtain a second query vector based on a third feature map and a first information interaction network, provided by some exemplary embodiments of the present disclosure. Fig.10 The method shown may include step 1010 and step 1020 .

[0139] Step 1010: Based on the second self-attention network in the first information interaction network, update the first query vector to obtain a seventh query vector.

[0140] Optionally, the first information interaction network may include a second self-attention network and a second cross-attention network; wherein the second self-attention network may be a network based on a self-attention mechanism, and the second cross-attention network may be a network based on a cross-attention mechanism.

[0141] Optionally, the specific implementation of step 1010 may refer to the above description of the specific implementation of step 910, which will not be repeated here.

[0142] Step 1020: Update the seventh query vector based on the third feature map and the second cross attention network in the first information interaction network to obtain a second query vector.

[0143] Optionally, the specific implementation of step 1020 may refer to the above description of the specific implementation of step 920, which will not be repeated here.

[0144] In the embodiments of the present disclosure, the first query vector can be updated through the self-attention mechanism, which is equivalent to obtaining a seventh query vector that is more accurate and effective than the first query vector through instance-level information interaction. Afterwards, the third feature map can be interacted with the seventh query vector through the cross-attention mechanism to learn useful information from the third feature map for updating the seventh query vector, thereby obtaining a second query vector that is more accurate and effective than the seventh query vector. Therefore, through the use of the self-attention mechanism and the cross-attention mechanism, the first query vector can be effectively optimized to obtain a reasonable and reliable second query vector.

[0145] It should be noted that the above describes the method for determining the detection result of the object to be detected based on the third feature map and the fourth feature map, and the method for determining the detection result of the object to be detected based on the third feature map and the fourth feature map is not limited to this. For example, after obtaining the third feature map and the fourth feature map, the third feature map can be updated to the fifth feature map through the self-attention mechanism, and the fourth feature map can be updated to the sixth feature map through the self-attention mechanism. Next, the second similarity between at least some pixels in the fifth feature map and at least some pixels in the sixth feature map can be determined. Based on the sixth feature map and the second similarity, the fifth feature map can be updated to the seventh feature map. Based on the fifth feature map and the second similarity, the sixth feature map can be updated to the eighth feature map. Afterwards, the detection result of the object to be detected can be determined based on the seventh feature map and the eighth feature map.

[0146] Optionally, based on the seventh characteristic map and the eighth characteristic map, Figure 6 The detection result of the object to be detected is determined in a similar manner to the embodiment shown. Alternatively, the seventh feature map and the eighth feature map can be updated again through the self-attention mechanism, and the subsequent process is similar to the process after obtaining the fifth feature map and the sixth feature map in the previous paragraph, which will not be repeated here.

[0147] In some optional examples, the detection result of the detected object determined in step 160 may include:

[0148] New information of a first number of objects to be detected;

[0149] and / or,

[0150] Disappearance information of a second number of objects to be detected.

[0151] In one example, based on the third feature map and the fourth feature map, the prediction network determines that there are 5 objects to be detected in the first image, namely, object 1 to be detected, object 2 to be detected, object 3 to be detected, object 4 to be detected, and object 5 to be detected. The prediction network also determines that there are 4 objects to be detected in the second image, namely, object 1 to be detected, object 2 to be detected, object 6 to be detected, and object 7 to be detected. Obviously, compared with the first image, two new objects to be detected appear in the second image, namely, object 6 to be detected and object 7 to be detected, and compared with the first image, three objects to be detected disappear in the second image, namely, object 3 to be detected, object 4 to be detected, and object 5 to be detected. Then, the detection result of the detected object determined in step 160 may include the newly added information of the two objects to be detected, indicating that the objects 6 to be detected and the objects 7 to be detected are newly added. The detection result of the detected object determined in step 160 may also include the disappearance information of the three objects to be detected, indicating that the objects 3 to be detected, the objects 4 to be detected, and the objects 5 to be detected disappear.

[0152] It should be noted that the newly added information of the first number of objects to be detected and the disappearance information of the second number of objects to be detected are both information about changes in the objects to be detected between the first image and the second image. That is, by adopting the embodiments of the present disclosure, it is possible to effectively detect objects to be detected that have changed (i.e., appeared or disappeared) between images, and then the target detection model has a good detection capability for objects to be detected that have changed. For example, the target detection module can accurately and stably detect objects to be detected that have changed.

[0153] In some optional examples, the same scene can be photographed by the same camera at different shooting times to obtain a first image and a second image. A second feature map can be obtained by extracting features from the second image. The second feature map can be a multi-scale feature map, for example, including Figure 3 The sub-feature maps 310, 320 and 330 in FIG. Similarly, by performing feature extraction on the first image, a first feature map can be obtained. The first feature map can belong to a multi-scale feature map, for example, including Figure 3 Sub-feature graph 340, sub-feature graph 350 and sub-feature graph 360 in.

[0154] Optionally, the first feature map can be represented as X_l, and the second feature map can be represented as X_r. Based on the self-attention mechanism, X_l can be updated to X_l', and X_r can be updated to X_r'. Based on the cross-attention mechanism, X_l' can be updated to X_l'', and X_r'' can be updated to X_r''. The process of obtaining X_l'' from X_l and obtaining X_r'' from X_r can be considered as one round of updating of the feature map. There can be a total of 6 rounds of feature map updates, and the result of X_l after 6 rounds of updates can be represented as X_l''' (which can be used as the third feature map above), and the result of X_r after 6 rounds of updates can be represented as X_r''' (which can be used as the fourth feature map above). Based on X_l''' and X_r''', the detection result of the object to be detected can be determined.

[0155] Figure 3 In the figure, the circular pattern is equivalent to the reference point mentioned above, the five-pointed star pattern is equivalent to the sampling point mentioned above, the arrowed line segment from X_l to X_l' and the arrowed line segment from X_r to X_r' represent the self-attention mechanism, and the arrowed line segment from X_l' to X_r'' and the arrowed line segment from X_r' to X_l'' represent the cross-attention mechanism.

[0156] It should be noted that in the process of determining the detection results of the object to be detected, based on the self-attention mechanism, information interaction within the graph can be realized, and based on the cross-attention mechanism, information interaction between graphs can be realized, thereby better ensuring the accuracy and reliability of the detection results of the object to be detected.

[0157] In some optional examples, the third feature map may belong to a multi-scale feature map, for example including Fig.11 , 1110, 1120, and 1130 in the sub-feature map. Similarly, the fourth feature map may belong to a multi-scale feature map, for example, including Fig.11 Sub-feature graph 1140, sub-feature graph 1150 and sub-feature graph 1160 in.

[0158] Optionally, a first query vector can be obtained through an initialization operation. Based on the third feature map and the fourth feature map, the first query vector can be updated several times through a self-attention mechanism and a cross-attention mechanism to obtain the second query vector, the fourth query vector, the third query vector, and the fifth query vector mentioned above. Based on the second query vector and the fourth query vector, a detection result of the object to be detected in the first image can be generated. Based on the third query vector and the fifth query vector, a detection result of the object to be detected in the second image can be generated.

[0159] Fig.11In the figure, the circular pattern is equivalent to the reference point mentioned above, the five-pointed star pattern is equivalent to the sampling point mentioned above, the triangle pattern represents the query vector, the arrowed line segment from the third feature map to the query vector and the arrowed line segment from the fourth feature map to the query vector represent the cross attention mechanism, the solid line with an arrow from the query vector to the query vector represents the self-attention mechanism, and the dashed line with an arrow from the query vector to the query vector represents the identity mapping. As introduced above, Figure 3 In the illustrated embodiment, the feature map may be updated in 6 rounds, that is, the feature map may be updated iteratively. Similarly, Fig.11 In the illustrated embodiment, the query vector may also be updated iteratively.

[0160] In summary, by adopting the embodiments of the present disclosure, it is possible to efficiently and reliably realize intra-image information interaction and inter-image information interaction, so as to optimize the representation of the query vector and obtain a query vector suitable for mutual decoding of two images (for example, a first image and a second image). On this basis, a complete, accurate and reliable target detection result can be obtained.

[0161] Exemplary Devices

[0162] Fig.12 It is a schematic diagram of the structure of a target detection device provided by some exemplary embodiments of the present disclosure. Fig.12 The apparatus shown may include:

[0163] An acquisition module 1210 is used to acquire a first image and a second image obtained by photographing the same scene; wherein the first image and the second image are photographed at different times and / or at different viewing angles;

[0164] The feature extraction module 1220 is used to extract features from the first image obtained by the acquisition module 1210 to obtain a first feature map, and to extract features from the second image obtained by the acquisition module 1210 to obtain a second feature map;

[0165] A first determination module 1230, configured to determine a first similarity between a feature of a first pixel in the first feature map obtained by the feature extraction module 1220 and a feature of a second pixel in the second feature map obtained by the feature extraction module 1220;

[0166] A first updating module 1240 is configured to update the feature of the first pixel in the first feature map obtained by the feature extraction module 1220 based on the second feature map obtained by the feature extraction module 1220 and the first similarity determined by the first determining module 1230 to obtain a third feature map;

[0167] A second updating module 1250 is configured to update the feature of the second pixel in the second feature map obtained by the feature extraction module 1220 based on the first feature map obtained by the feature extraction module 1220 and the first similarity determined by the first determination module 1230 to obtain a fourth feature map;

[0168] The second determination module 1260 is used to determine the detection result of the object to be detected based on the third feature map obtained by the first updating module 1240 and the fourth feature map obtained by the second updating module 1250.

[0169] In some optional examples, such as Fig.13 As shown, the first update module 1240 includes:

[0170] A selection submodule 1310 is used to select a target second pixel point that satisfies a preset feature similarity condition with the first pixel point from each second pixel point in the second feature map obtained by the feature extraction module 1220 based on the first similarity determined by the first determination module 1230;

[0171] The first updating submodule 1320 is used to update the feature of the first pixel point based on the point information of the target second pixel point selected by the selecting submodule 1310.

[0172] In some optional examples, such as Figure 14-1 As shown, the first updating submodule 1320 includes:

[0173] A first determining unit 1402 is used to determine the coordinates and weight of each second pixel point in the second feature map obtained by the feature extraction module 1220, based on the point information of the target second pixel point selected by the selection submodule 1310, of each second pixel point in the plurality of second pixel points that meet the preset sampling condition;

[0174] The first updating unit 1404 is configured to update the feature of the first pixel point based on the coordinates and weight of each second pixel point that meets the preset sampling condition and is determined by the first determining unit 1402 .

[0175] In some optional examples, the second feature map includes a first-scale sub-feature map and a second-scale sub-feature map;

[0176] like Figure 14-2 As shown, the selection submodule 1310 includes:

[0177] A first selection unit 1410 is configured to select, from each second pixel point in the first scale sub-feature map, a target second pixel point that satisfies a preset feature similarity condition with the first pixel point based on the first similarity determined by the first determination module 1230;

[0178] A second determining unit 1420 is used to determine a size ratio relationship between the second scale sub-feature map and the first scale sub-feature map;

[0179] A mapping unit 1430 is used to map the coordinates of the second pixel point of the target in the first scale sub-feature map selected by the first selection unit 1410 to the second scale sub-feature map based on the size ratio relationship determined by the second determination unit 1420, so as to obtain the mapping coordinates on the second scale sub-feature map;

[0180] The second selection unit 1440 is used to select another target second pixel point that satisfies a preset feature similarity condition with the first pixel point from each second pixel point in the second scale sub-feature map based on the mapping coordinates obtained by the mapping unit 1430.

[0181] In some optional examples, such as Fig.15 As shown, the second determination module 1260 includes:

[0182] A second updating submodule 1510 is used to update the first query vector of the object to be detected obtained through the initialization operation based on the third feature map obtained by the first updating module 1240 and the first information interaction network to obtain a second query vector;

[0183] A third updating submodule 1520 is used to update the first query vector based on the fourth feature graph obtained by the second updating module 1250 and the first information interaction network to obtain a third query vector;

[0184] The generation submodule 1530 is used to generate a detection result of the object to be detected based on the second query vector obtained by the second update submodule 1510, the third query vector obtained by the third update submodule 1520, the third feature map obtained by the first update module 1240, the fourth feature map obtained by the second update module 1250, the second information interaction network and the prediction network.

[0185] In some optional examples, such as Fig.16 As shown, the generating submodule 1530 includes:

[0186] A second updating unit 1610 is configured to update the third query vector obtained by the third updating submodule 1520 based on the third feature graph and the second information interaction network obtained by the first updating module 1240 to obtain a fourth query vector;

[0187] A first generating unit 1620, configured to generate a detection result of the object to be detected in the first image via a prediction network based on the second query vector obtained by the second updating submodule 1510 and the fourth query vector obtained by the second updating unit 1610;

[0188] A third updating unit 1630 is configured to update the second query vector obtained by the second updating submodule 1510 based on the fourth feature graph and the second information interaction network obtained by the second updating module 1250 to obtain a fifth query vector;

[0189] The second generating unit 1640 is used to generate a detection result of the object to be detected in the second image via a prediction network based on the third query vector obtained by the third updating submodule 1520 and the fifth query vector obtained by the third updating unit 1630.

[0190] In some optional examples, such as Fig.17 As shown, the second updating unit 1610 includes:

[0191] A first updating subunit 1710 is used to update the third query vector obtained by the third updating submodule 1520 based on the first self-attention network in the second information interaction network to obtain a sixth query vector;

[0192] The second updating subunit 1720 is used to update the sixth query vector obtained by the first updating subunit 1710 based on the third feature map obtained by the first updating module 1240 and the first cross attention network in the second information interaction network to obtain a fourth query vector.

[0193] In some optional examples, such as Fig.18 As shown, the second updating submodule 1510 includes:

[0194] A fourth updating unit 1810 is used to update the first query vector based on the second self-attention network in the first information interaction network to obtain a seventh query vector;

[0195] The fifth updating unit 1820 is used to update the seventh query vector obtained by the fourth updating unit 1810 based on the third feature map obtained by the first updating module 1240 and the second cross attention network in the first information interaction network to obtain a second query vector.

[0196] In some optional examples, the detection results include:

[0197] New information of a first number of objects to be detected;

[0198] and / or,

[0199] Disappearance information of a second number of objects to be detected.

[0200] In the device of the present disclosure, the various optional embodiments, optional implementation methods and optional examples disclosed above can be flexibly selected and combined as needed to achieve corresponding functions and effects, and the present disclosure does not list them one by one.

[0201] Exemplary Electronic Devices

[0202] Fig.19 A block diagram of an electronic device according to an embodiment of the present disclosure is illustrated. The electronic device 1900 includes one or more processors 1910 and a memory 1920 .

[0203] The processor 1910 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 1900 to perform desired functions.

[0204] The memory 1920 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 1910 may execute one or more computer program instructions to implement the methods of the various embodiments of the present disclosure described above and / or other desired functions.

[0205] In one example, the electronic device 1900 may further include: an input device 1930 and an output device 1940 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0206] The input device 1930 may also include, for example, a keyboard, a mouse, etc.

[0207] The output device 1940 can output various information to the outside, and may include, for example, a display, a speaker, a printer, a communication network and its connected remote output device, etc.

[0208] Of course, to simplify, Fig.19 Only some of the components related to the present disclosure in the electronic device 1900 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application scenarios, the electronic device 1900 may also include any other appropriate components.

[0209] Exemplary computer program products and computer-readable storage media

[0210] In addition to the above-mentioned methods and devices, an embodiment of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the method according to various embodiments of the present disclosure described in the above-mentioned "Exemplary Method" section of this specification.

[0211] The computer program product may be written in any combination of one or more programming languages ​​to write program code for performing the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0212] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, enables the processor to execute the steps of the method according to various embodiments of the present disclosure described in the above “Exemplary Method” section of this specification.

[0213] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0214] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations. It cannot be considered that these advantages, strengths, effects, etc. must be possessed by each embodiment of the present disclosure. The specific details disclosed above are only for the purpose of illustration and ease of understanding, not limitation. The above details do not limit the present disclosure to being implemented by adopting the above specific details.

[0215] Those skilled in the art may make various changes and modifications to the present disclosure without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the present disclosure claims and their equivalents, the present disclosure is also intended to include these modifications and variations.

Claims

1. A target detection method, comprising: Acquire a first image and a second image obtained by photographing the same scene; wherein the first image and the second image are photographed at different times and / or at different viewing angles; Performing feature extraction on the first image to obtain a first feature map, and performing feature extraction on the second image to obtain a second feature map; Determine a first similarity between a feature of a first pixel in the first feature map and a feature of a second pixel in the second feature map; Based on the second feature map and the first similarity, updating the feature of the first pixel in the first feature map to obtain a third feature map; Based on the first feature map and the first similarity, updating the feature of the second pixel in the second feature map to obtain a fourth feature map; Based on the third feature map and the fourth feature map, a detection result of the object to be detected is determined.

2. The method according to claim 1, wherein: The updating the feature of the first pixel in the first feature map based on the second feature map and the first similarity includes: Based on the first similarity, selecting a target second pixel point that satisfies a preset feature similarity condition with the first pixel point from each second pixel point in the second feature map; Based on the point information of the target second pixel point, the feature of the first pixel point is updated.

3. The method according to claim 2, wherein: The updating of the feature of the first pixel point based on the point information of the second pixel point of the target includes: Based on the point information of the target second pixel point, determine the coordinates and weight of each second pixel point in the second feature map, among the plurality of second pixel points that meet the preset sampling condition; Based on the coordinates and weight of each second pixel point that meets the preset sampling condition, the features of the first pixel point are updated.

4. The method according to claim 2, wherein: The second feature map includes a first scale sub-feature map and a second scale sub-feature map; The selecting, based on the first similarity, from each second pixel in the second feature map, a target second pixel that satisfies a preset feature similarity condition with the first pixel, comprises: Based on the first similarity, selecting a target second pixel point that satisfies a preset feature similarity condition with the first pixel point from each second pixel point in the first-scale sub-feature map; Determining a size ratio relationship between the second-scale sub-feature map and the first-scale sub-feature map; Based on the size ratio relationship, mapping the coordinates of the second pixel point of the target in the first scale sub-feature map to the second scale sub-feature map to obtain the mapping coordinates on the second scale sub-feature map; Based on the mapping coordinates, another target second pixel point that satisfies a preset feature similarity condition with the first pixel point is selected from each second pixel point in the second-scale sub-feature map.

5. The method according to any one of claims 1 to 4, wherein: The determining, based on the third feature map and the fourth feature map, a detection result of the object to be detected includes: Based on the third feature map and the first information interaction network, updating the first query vector of the object to be detected obtained through the initialization operation to obtain a second query vector; Based on the fourth feature graph and the first information interaction network, updating the first query vector to obtain a third query vector; A detection result of the object to be detected is generated based on the second query vector, the third query vector, the third feature map, the fourth feature map, the second information interaction network and the prediction network.

6. The method according to claim 5, wherein: The generating a detection result of the object to be detected based on the second query vector, the third query vector, the third feature map, the fourth feature map, the second information interaction network and the prediction network includes: Based on the third feature graph and the second information interaction network, updating the third query vector to obtain a fourth query vector; Based on the second query vector and the fourth query vector, generating a detection result of the object to be detected in the first image via the prediction network; Based on the fourth feature graph and the second information interaction network, updating the second query vector to obtain a fifth query vector; Based on the third query vector and the fifth query vector, a detection result of the object to be detected in the second image is generated via the prediction network.

7. The method according to claim 6, wherein: The updating the third query vector based on the third feature graph and the second information interaction network to obtain a fourth query vector includes: Based on the first self-attention network in the second information interaction network, updating the third query vector to obtain a sixth query vector; Based on the third feature map and the first cross attention network in the second information interaction network, the sixth query vector is updated to obtain a fourth query vector.

8. The method according to claim 5, wherein: The updating of the first query vector of the object to be detected obtained by the initialization operation based on the third feature map and the first information interaction network to obtain the second query vector includes: Based on the second self-attention network in the first information interaction network, updating the first query vector to obtain a seventh query vector; Based on the third feature map and the second cross attention network in the first information interaction network, the seventh query vector is updated to obtain a second query vector.

9. The method according to any one of claims 1 to 4, wherein: The test results include: New information of a first number of the objects to be detected; and / or, Disappearance information of a second number of the objects to be detected.

10. A target detection device, comprising: An acquisition module, used to acquire a first image and a second image obtained by photographing the same scene; wherein the first image and the second image are photographed at different times and / or at different viewing angles; a feature extraction module, configured to extract features from the first image obtained by the acquisition module to obtain a first feature map, and to extract features from the second image obtained by the acquisition module to obtain a second feature map; A first determination module, used to determine a first similarity between a feature of a first pixel point in the first feature map obtained by the feature extraction module and a feature of a second pixel point in the second feature map obtained by the feature extraction module; A first updating module, configured to update the feature of the first pixel in the first feature map obtained by the feature extraction module based on the second feature map obtained by the feature extraction module and the first similarity determined by the first determination module, so as to obtain a third feature map; A second updating module, configured to update the feature of the second pixel in the second feature map obtained by the feature extraction module based on the first feature map obtained by the feature extraction module and the first similarity determined by the first determination module, so as to obtain a fourth feature map; The second determination module is used to determine the detection result of the object to be detected based on the third feature map obtained by the first updating module and the fourth feature map obtained by the second updating module.

11. A computer-readable storage medium storing a computer program, wherein the computer program is used to execute the target detection method according to any one of claims 1 to 9.

12. An electronic device, comprising: processor; a memory for storing instructions executable by the processor; The processor is used to read the executable instructions from the memory and execute the instructions to implement the target detection method described in any one of claims 1-9 above.

Images