Target object detection method and device, storage medium and electronic equipment

Abstract

The invention discloses a target object detection method and device, a storage medium and electronic equipment. The method comprises the following steps: acquiring a first image and a second image obtained by shooting the same scene; wherein the shooting moments and / or shooting viewing angles of the first image and the second image are different; performing feature extraction on the first image to obtain a first feature map, and performing feature extraction on the second image to obtain a second feature map; based on the first feature map, the second feature map and the first information interaction network, updating a first query vector of the to-be-detected object to obtain a second query vector; updating the second query vector based on the second feature map and the second information interaction network to obtain a third query vector; and generating a first detection result of the to-be-detected object based on the third query vector and the prediction network. According to the embodiment of the invention, the target detection precision of the image can be well ensured.

Description

Technical Field

[0001] The present disclosure relates to machine vision technology, and in particular to a method, device, storage medium and electronic device for detecting a target object. Background Art

[0002] In some scenarios, it is necessary to perform object detection on an image, for example, to find objects of interest in an image, determine the position and size of these objects, etc. How to ensure the accuracy of object detection in an image is a problem worthy of attention for those skilled in the art. Summary of the invention

[0003] In order to solve the above technical problems, the present disclosure provides a target detection method, device, storage medium and electronic device.

[0004] According to one aspect of an embodiment of the present disclosure, a method for detecting a target object is provided, including:

[0005] Acquire a first image and a second image obtained by photographing the same scene; wherein the first image and the second image are photographed at different times and / or at different viewing angles;

[0006] Performing feature extraction on the first image to obtain a first feature map, and performing feature extraction on the second image to obtain a second feature map;

[0007] Based on the first feature map, the second feature map and the first information interaction network, updating a first query vector of the object to be detected to obtain a second query vector;

[0008] Based on the second feature graph and the second information interaction network, updating the second query vector to obtain a third query vector;

[0009] Based on the third query vector and the prediction network, a first detection result of the object to be detected is generated.

[0010] According to another aspect of the embodiments of the present disclosure, there is provided a device for detecting a target object, comprising:

[0011] An acquisition module, used to acquire a first image and a second image obtained by photographing the same scene; wherein the first image and the second image are photographed at different times and / or at different viewing angles;

[0012] a feature extraction module, configured to extract features from the first image acquired by the acquisition module to obtain a first feature map, and to extract features from the second image acquired by the acquisition module to obtain a second feature map;

[0013] A first updating module, configured to update a first query vector of the object to be detected based on the first feature map and the second feature map obtained by the feature extraction module and a first information interaction network to obtain a second query vector;

[0014] A second updating module, configured to update the second query vector obtained by the first updating module based on the second feature graph and the second information interaction network obtained by the feature extraction module to obtain a third query vector;

[0015] The first generating module is used to generate a first detection result of the object to be detected based on the third query vector and the prediction network obtained by the second updating module.

[0016] According to another aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and the computer program is used to execute the above-mentioned target object detection method.

[0017] According to another aspect of the embodiments of the present disclosure, an electronic device is provided, the electronic device comprising:

[0018] processor;

[0019] a memory for storing instructions executable by the processor;

[0020] The processor is used to read the executable instructions from the memory and execute the instructions to implement the above-mentioned target object detection method.

[0021] According to another aspect of the embodiments of the present disclosure, a computer program product is provided. When instructions in the computer program product are executed by a processor, the above-mentioned target object detection method is performed.

[0022] Based on the target object detection method, device, storage medium, electronic device and program product provided by the above embodiments of the present disclosure, the first image and the second image are images obtained by shooting the same scene, and the shooting time and / or shooting angle of the first image and the second image are different. Then, the information carried by each of the first image and the second image has both the same part and different parts. By extracting features from the first image, a first feature map can be obtained. By extracting features from the second image, a second feature map can be obtained. Since the information carried by each of the first image and the second image has both the same part and different parts, the features included in each of the first feature map and the second feature map are different. On the basis of these features and the first query vector, the first information interaction network and the second information interaction network are used, and through information interaction, the second query vector and the third query vector can be obtained in turn. Compared with the first query vector, the second query vector is more accurate and effective, and the third query vector is more accurate and effective than the second query vector. In this way, based on the third query vector and the prediction network, the first detection result of the object to be detected is generated, which can better guarantee the accuracy and reliability of the first detection result. Therefore, the embodiment of the present disclosure can better guarantee the target detection accuracy of the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a flowchart of a method for detecting a target object provided by some exemplary embodiments of the present disclosure.

[0024] Figure 2 is a schematic diagram of the attention mechanism in some exemplary embodiments of the present disclosure.

[0025] Figure 3 It is a flowchart of a method for updating a first query vector of an object to be detected to obtain a second query vector based on a first feature map, a second feature map and a first information interaction network, provided by some exemplary embodiments of the present disclosure.

[0026] Figure 4 It is a flowchart of a method for updating the feature of a first pixel in a first feature map based on a second feature map and a similarity, provided by some exemplary embodiments of the present disclosure.

[0027] Figure 5 It is a schematic diagram of some exemplary embodiments of the present disclosure for realizing target detection.

[0028] Figure 6 It is a flowchart of a method for updating the features of a first pixel point based on point information of a target second pixel point provided by some exemplary embodiments of the present disclosure.

[0029] Figure 7It is a flowchart of a method for selecting a target second pixel point that satisfies a preset feature similarity condition with a first pixel point from each second pixel point in a second feature map based on similarity, provided by some exemplary embodiments of the present disclosure.

[0030] Figure 8 It is a flowchart of a method for detecting a target object provided by some other exemplary embodiments of the present disclosure.

[0031] Fig. 9 It is a flowchart of a method for updating a first query vector based on a second feature graph and utilizing a first information interaction network to obtain a fourth query vector, provided by some exemplary embodiments of the present disclosure.

[0032] Fig.10 It is a flowchart of a method for detecting a target object provided by some further exemplary embodiments of the present disclosure.

[0033] Fig.11 It is a flowchart of a method for updating a second query vector to obtain a third query vector based on a second feature map and a second information interaction network provided by some exemplary embodiments of the present disclosure.

[0034] Fig.12 It is a schematic diagram of some other exemplary embodiments of the present disclosure for realizing target detection.

[0035] Fig.13 It is a schematic diagram of the structure of a target object detection device provided by some exemplary embodiments of the present disclosure.

[0036] Figure 14-1 is a schematic diagram of modules related to the generation of the first detection result in some exemplary embodiments of the present disclosure.

[0037] Figure 14-2 It is a schematic diagram of modules related to the determination of newly added information and / or disappeared information of an object to be detected in some exemplary embodiments of the present disclosure.

[0038] Fig.15 is a schematic diagram of the structure of the first update module in some exemplary embodiments of the present disclosure.

[0039] Fig.16 is a schematic diagram of the structure of the first updating submodule in some exemplary embodiments of the present disclosure.

[0040] Fig.17 is a schematic diagram of the structure of an update unit in some exemplary embodiments of the present disclosure.

[0041] Fig.18 It is a schematic diagram of the structure of selected units in some exemplary embodiments of the present disclosure.

[0042] Fig.19It is a schematic diagram of the structure of an electronic device provided by some exemplary embodiments of the present disclosure. DETAILED DESCRIPTION

[0043] To explain the present disclosure, example embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments. It should be understood that the present disclosure is not limited to the example embodiments.

[0044] It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.

[0045] Application Overview

[0046] Image object detection technology can be applied to many fields. For example, image object detection technology can be applied to the industrial field to assist in defect detection. For another example, image object detection technology can be applied to the medical field to assist in lesion detection. For another example, image object detection technology can be applied to the driving field to assist in environmental perception.

[0047] How to ensure the accuracy of target detection in images is a problem worthy of attention for those skilled in the art.

[0048] Exemplary Systems

[0049] In an embodiment of the present disclosure, the same scene can be photographed to obtain a first image and a second image with different shooting times and / or shooting angles, so that the information carried by the first image and the second image each has both the same part and different parts. By performing feature extraction on the first image, a first feature map can be obtained. By performing feature extraction on the second image, a second feature map can be obtained. In addition, a query vector of the object to be detected can be initialized. The query vector of the object to be detected can be a vector carrying relevant information of the object to be detected. The relevant information of the object to be detected can include, for example, but is not limited to the position, size, etc. of the object to be detected. Based on the first feature map and the second feature map, the query vector can be optimized through intra-graph information interaction and inter-graph information interaction, and based on the optimized query vector, the target detection results of the first image and the second image can be generated. In this way, by adopting the embodiment of the present disclosure, the target detection accuracy of the image can be better guaranteed.

[0050] Exemplary Methods

[0051] Figure 1 It is a flowchart of a method for detecting a target object provided by some exemplary embodiments of the present disclosure. Figure 1The method shown may include step 110 , step 120 , step 130 , step 140 , and step 150 .

[0052] Step 110, acquiring a first image and a second image obtained by photographing the same scene; wherein the first image and the second image are photographed at different times and / or at different viewing angles.

[0053] Optionally, the first image and the second image may be images taken successively by the same camera installed in the vehicle when the vehicle is stationary, and accordingly, the shooting time of the first image and the shooting time of the second image may be different, and the shooting angle of the first image and the shooting angle of the second image may be the same. Alternatively, the first image and the second image may be images taken successively by the same camera installed in the vehicle when the vehicle is turning, and accordingly, the shooting time of the first image and the shooting time of the second image may be different, and the shooting angle of the first image and the shooting angle of the second image may also be different. Alternatively, the first image and the second image may be images taken simultaneously by different cameras installed in the vehicle, and accordingly, the shooting time of the first image and the shooting time of the second image may be the same, and the shooting angle of the first image and the shooting angle of the second image may be different.

[0054] In one example, the vehicle may be provided with six cameras, namely, a first camera disposed in front of the vehicle, a second camera disposed in front of the left side of the vehicle, a third camera disposed in front of the right side of the vehicle, a fourth camera disposed in front of the vehicle, a fifth camera disposed in the rear left side of the vehicle, and a sixth camera disposed in the rear right side of the vehicle. At time t1 and time t2 different from time t1, the first camera may shoot the road scene, the image shot by the first camera at time t1 may be used as the first image, and the image shot by the first camera at time t2 may be used as the second image. Alternatively, at time t1, the first camera and the second camera may both shoot the road scene, the image shot by the first camera at time t1 may be used as the first image, and the image shot by the second camera at time t1 may be used as the second image.

[0055] Optionally, the vehicle on which the camera for capturing the first image and the second image is disposed may also be referred to as the host vehicle hereinafter.

[0056] Step 120: extract features from the first image to obtain a first feature map, and extract features from the second image to obtain a second feature map.

[0057] Optionally, a feature extractor may be used to perform feature extraction on the first image to obtain a first feature map. Similarly, a feature extractor may be used to perform feature extraction on the second image to obtain a second feature map. The feature extractor may be, for example, a feature extraction network in a pre-trained target detection model. The size of the first feature map and the size of the second feature map may be the same or different. For ease of understanding, the following description will take the case where the size of the first feature map and the size of the second feature map are the same as an example.

[0058] Step 130: Based on the first feature map, the second feature map and the first information interaction network, update the first query vector of the object to be detected to obtain a second query vector.

[0059] Optionally, the first information interaction network may be a network used for information interaction in the target detection model. The first information interaction network may be a network based on an attention mechanism. The attention mechanism may include, but is not limited to, a self-attention mechanism, a cross-attention mechanism, and the like. The self-attention mechanism may, for example, be a multi-scale deformable self-attention mechanism. The cross-attention mechanism may, for example, be a multi-scale deformable cross-attention mechanism. The network based on the attention mechanism may, for example, be a Transformer network, and the Transformer network may adopt an encoding-decoding architecture.

[0060] In general, the principle of the attention mechanism is: calculate the attention distribution on the given information (i.e. determine what information is important and what information is not important); perform weighted average operation based on the attention distribution to obtain the attention calculation result. Figure 2 , specifically: Source includes a series of<Key,Value> Data pairs (also called key-value pairs), for the element Query, calculate the similarity between the Query and each Key, get the weight corresponding to each Key, use these weights to perform weighted summation on each Value to get the final Attention calculation result (ie Attention Value).

[0061] Optionally, a first query vector of the object to be detected can be generated through an initialization operation. The initialization operation can be, for example, a random initialization operation. The objects to be detected can include, but are not limited to, pedestrians, cyclists, other vehicles other than the vehicle, etc. The first query vector can be a vector carrying relevant information of the object to be detected. The number of first query vectors can be multiple, for example, it can be V, and the value of V can be determined based on the maximum number of objects to be detected that may appear in a single image. For example, a maximum of 100 objects to be detected may appear in a single image, then V can be 100.

[0062] Optionally, based on the first feature map and the second feature map, the first information interaction network can be used to update the first query vector using an attention mechanism, thereby obtaining a second query vector that is more accurate and effective than the first query vector. For example, the second query vector can more accurately and effectively reflect the relevant information of the object to be detected. The number of second query vectors can also be V.

[0063] Step 140: Based on the second feature map and the second information interaction network, update the second query vector to obtain a third query vector.

[0064] Optionally, the second information interaction network may be a network used for information interaction in the target detection model. The second information interaction network may be a network based on an attention mechanism.

[0065] Optionally, the second feature map and the second query vector can be used as input data of the second information interaction network, and the second information interaction network can perform operations based on the input data to apply the attention mechanism to update the second query vector, thereby obtaining a third query vector that is more accurate and effective than the second query vector. For example, the third query vector can more accurately and effectively reflect the relevant information of the object to be detected. The number of third query vectors can also be V.

[0066] Step 150: Generate a first detection result of the object to be detected based on the third query vector and the prediction network.

[0067] Optionally, the prediction network may be a network in a target detection model for generating target detection results based on a given query vector.

[0068] Optionally, at least V third query vectors may be used as input data of the prediction network, and the prediction network may decode the input data to obtain a first detection result of the object to be detected. The first detection result may be used as the target detection result of the second image. The first detection result may be used to characterize which objects to be detected exist in the second image, and / or the position, size, etc. of these objects to be detected in the second image.

[0069] In the embodiment of the present disclosure, the first image and the second image are images obtained by shooting the same scene, and the shooting time and / or shooting angle of the first image and the second image are different. Then, the information carried by each of the first image and the second image has both the same part and different parts. By extracting features from the first image, a first feature map can be obtained. By extracting features from the second image, a second feature map can be obtained. Since the information carried by each of the first image and the second image has both the same part and different parts, the features included in each of the first feature map and the second feature map are different. Based on these features and the first query vector, the first information interaction network and the second information interaction network are used to obtain the second query vector and the third query vector in turn through information interaction. The second query vector is more accurate and effective than the first query vector, and the third query vector is more accurate and effective than the second query vector. In this way, based on the third query vector and the prediction network, the first detection result of the object to be detected is generated, which can better guarantee the accuracy and reliability of the first detection result. Therefore, the embodiment of the present disclosure can better guarantee the target detection accuracy of the image.

[0070] Figure 3 It is a flowchart of a method for updating a first query vector of an object to be detected to obtain a second query vector based on a first feature map, a second feature map and a first information interaction network, provided by some exemplary embodiments of the present disclosure. Figure 3 The method shown may include step 310 , step 320 , and step 330 .

[0071] Step 310: Determine the similarity between the feature of the first pixel in the first feature map and the feature of the second pixel in the second feature map.

[0072] Optionally, there may be a total of M pixels in the first feature map, and each of the M pixels may be used as a first pixel, that is, the first feature map may include M first pixels. Similarly, there may be a total of M pixels in the second feature map, and each of the M pixels may be used as a second pixel, that is, the second feature map may include M second pixels. Each of the M first pixels included in the first feature map and each of the M second pixels included in the second feature map may have corresponding features, respectively, and the features of any pixel are used to implicitly represent certain semantic information.

[0073] Optionally, the similarity between the features of at least some of the M first pixels and the features of at least some of the M second pixels can be determined according to a preset similarity algorithm, and the determined similarities can form a similarity set. The preset similarity algorithm can be, for example, a cosine similarity algorithm.

[0074] In one example, for each of the M first pixel points, the similarity between the feature of the first pixel point and the features of the M second pixel points can be determined to obtain M similarities corresponding to the first pixel point. In this way, M similarities corresponding to each of the M first pixel points can be obtained, so that a total of M*M similarities can be obtained. M*M similarities can form a similarity set.

[0075] Step 320: Based on the second feature map and the similarity, update the feature of the first pixel in the first feature map to obtain a third feature map.

[0076] Optionally, based on the second feature map and the similarity set, the features of the M first pixels included in the first feature map can be updated to update the first feature map to a third feature map. The size of the third feature map can be the same as that of the first feature map.

[0077] Step 330: Based on the third feature graph and the first information interaction network, update the first query vector to obtain a second query vector.

[0078] Optionally, the third feature map and the first query vector can be used as input data of the first information interaction network, and the first information interaction network can perform operations based on the input data to apply the attention mechanism to update the first query vector, thereby obtaining a second query vector that is more accurate and effective than the first query vector. For example, the second query vector can more accurately and effectively reflect the relevant information of the object to be detected. The number of second query vectors can also be V.

[0079] In an embodiment of the present disclosure, after determining the similarity between the feature of the first pixel in the first feature map and the feature of the second pixel in the second feature map, the feature of the first pixel in the first feature map can be updated based on the second feature map and the similarity, which is equivalent to performing pixel-level feature interaction on the first feature map and the second feature map, thereby obtaining a third feature map that is more accurate and effective than the first feature map. For example, the features in the third feature map can more accurately and effectively represent the relevant information of the object to be detected. In addition, based on the first information interaction network, the information interaction between the third feature map and the first query vector can be effectively realized to obtain a second query vector that is more accurate and effective than the first query vector. On the basis of the second query vector, the third query vector and the first detection result are determined in sequence, which can better ensure the accuracy and reliability of the first detection result.

[0080] Figure 4 It is a flowchart of a method for updating the feature of a first pixel in a first feature map based on a second feature map and a similarity, provided by some exemplary embodiments of the present disclosure. Figure 4 The illustrated method may include step 410 and step 420 .

[0081] Step 410 , based on the similarity, select a target second pixel point that satisfies a preset feature similarity condition with the first pixel point from each second pixel point in the second feature map.

[0082] Optionally, the preset feature similarity condition may include the similarity between features being ranked first (specifically for the case where multiple similarities are ranked from first to last), or the similarity between features being greater than a preset similarity. As an example, the preset similarity may be 80%, 85%, 90%, etc., which are not listed here one by one. It should be noted that the features involved in this paragraph refer to the features of the pixel points, including the semantic information of the feature map.

[0083] Optionally, the second feature map may belong to a multi-scale feature map, then the second feature map may include multiple sub-feature maps of different scales, and the multiple sub-feature maps may be represented as R sub-feature maps; wherein R may be an integer greater than or equal to 2. For example, the R sub-feature maps may be 3 sub-feature maps, such as Figure 5 As shown, the three sub-feature graphs may be sub-feature graph 510, sub-feature graph 520 and sub-feature graph 530 in sequence.

[0084] For a certain sub-feature map among the R sub-feature maps, a target second pixel point that satisfies the preset feature similarity condition with the first pixel point can be selected from each second pixel point in the sub-feature map. Assuming that there are a total of M1 second pixel points in the sub-feature map, the similarities between the features of the first pixel point and the respective features of the M1 second pixel points can be obtained from the similarity set to obtain M1 similarities that correspond one to one with the M1 second pixel points. The M1 similarities can be arranged in order from large to small, and then the similarity with the highest ranking can be selected (or the second highest ranking similarity can be selected), and the second pixel point corresponding to the selected similarity can be used as the target second pixel point in the sub-feature map that satisfies the preset feature similarity condition with the first pixel point. In this way, the similarities with R sub-features can be obtained. Figure 1 A corresponding R target second pixel points.

[0085] The above describes the case where there are multiple target second pixel points. In a specific implementation, the number of target second pixel points may also be only one. For example, the second feature map may be a single-scale feature map, and the similarities between the features of the first pixel point and the features of the M second pixel points in the second feature map may be obtained from the similarity set to obtain M similarities corresponding to the M second pixel points. The M similarities may be arranged in order from large to small, and the similarity with the highest ranking may be selected, and the second pixel point corresponding to the selected similarity may be used as the only target second pixel point.

[0086] Step 420: Update the feature of the first pixel based on the point information of the target second pixel.

[0087] Optionally, the point information of the second target pixel may include but is not limited to the coordinates, features, etc. of the second target pixel. The coordinates of the second target pixel are used to characterize the pixel position of the second target pixel. The features of the second target pixel include semantic information of the feature map. The number of the second target pixel may be one or more, and updating the features of the first pixel based on the point information of the second target pixel may include: updating the features of the first pixel based on the point information of one or more second target pixels.

[0088] In some optional embodiments of the present disclosure, Figure 6 As shown, step 420 may include step 610 and step 620. For ease of understanding, Figure 6 In the illustrated implementation manner, a case where the number of target second pixel points is one is taken as an example for description.

[0089] Step 610, based on the point information of the target second pixel point, determine the coordinates and weight of each second pixel point in the second feature map, among multiple second pixel points that meet the preset sampling condition.

[0090] Optionally, the preset sampling condition includes pixels distributed within a preset range around the second pixel of the target, and / or pixels with a high feature similarity to the second pixel of the target. Based on the point information of the second pixel of the target, the coordinates and weights of each second pixel in the plurality of second pixels that meet the preset sampling condition can be determined by the sampling point determination network in the target detection model. For example, the sampling point determination network is used to predict the coordinates and weights of each of the plurality of sampling points according to the coordinates and features of the reference point; wherein the reference point and the plurality of sampling points can be different pixels located in the same image. The point information of the second pixel of the target may include the coordinates and features of the second pixel of the target, and the coordinates and features of the second pixel of the target can be used as input data of the sampling point determination network, and the sampling point determination network can perform operations accordingly to output the coordinates and weights of each of the plurality of sampling points, and each of the plurality of sampling points can be used as a second pixel in the second feature map that meets the preset sampling condition. In this way, the sampling point determination network can be used to efficiently and quickly determine the coordinates and weights of each of the plurality of second pixels that meet the preset sampling condition.

[0091] In some embodiments, the position of the target second pixel point can be used as the center to determine a target area with a set size, and random sampling or regular sampling of pixels can be performed in the target area to obtain multiple sampling points, each of which can be used as a second pixel point that meets the preset sampling condition in the second feature map. Optionally, the target area can be a rectangular area, and accordingly, the set size can include length and width. Alternatively, the target area can be a circular area, and accordingly, the set size can include radius or diameter. In addition, for each of the multiple sampling points, the weight of the sampling point can be determined based on the distance between the position of the sampling point and the position of the target second pixel point, and the weight of the sampling point is the weight of the corresponding second pixel point that meets the preset sampling condition. Optionally, when determining the weight based on the distance, the weight can be negatively correlated with the distance.

[0092] Step 620: Update the features of the first pixel based on the coordinates and weights of each second pixel that meets the preset sampling condition.

[0093] Optionally, the weights of the respective second pixel points satisfying the preset sampling condition may be used to perform weighted averaging on the respective coordinates of the respective second pixel points satisfying the preset sampling condition to obtain weighted coordinates.

[0094] In one example, the plurality of second pixel points satisfying the preset sampling condition are N second pixel points, the N second pixel points correspond one-to-one to the N coordinates, the N second pixel points also correspond one-to-one to the N weights, and the N coordinates are sequentially represented as P1, P2, P3, ..., PN , the N weights are represented as K1, K2, K3, ..., K N , the weighted coordinate is expressed as P, then:

[0095] P=(K1*P1+K2*P2+K3*P3+……+K N *P N ) / (K1+K2+K3+……+K N )

[0096] After obtaining the weighted coordinates, the features of the second pixel point having the weighted coordinates in the second feature map may be determined (for ease of description, it may be referred to as the target feature hereinafter), and the features of the first pixel point may be updated based on the target feature. For example, the features of the first pixel point may be updated to the target feature. For another example, a mean operation or a weighted average operation may be performed on the target feature and the features of the first pixel point, and the features of the first pixel point may be updated to the mean operation result or the weighted average operation result.

[0097] In the above implementation, based on the point information of the target second pixel point, the coordinates and weights of each second pixel point in the second feature map and the multiple second pixel points that meet the preset sampling conditions can be determined efficiently and quickly. On this basis, the features of the first pixel point can be updated efficiently and quickly by using the weighted average operation. In this way, the first feature map can be updated to the third feature map through the pixel-level feature interaction between the first feature map and the second feature map. Since the update of the features of the first pixel point uses the coordinates and weights of a specific number of second pixel points, the data referenced by the update of the features of the first pixel point is very rich, which can avoid the adverse effects that may be caused by only referring to individual pixels, and is conducive to ensuring the pixel-level feature interaction effect.

[0098] In some embodiments, the number of target second pixel points is multiple, for example, R, then for each of the R target second pixel points, the above method can be used to obtain the corresponding N coordinates and N weights, and based on the corresponding N coordinates and N weights, the corresponding target feature is obtained. Next, the feature of the first pixel point can be updated based on the R target features that correspond one-to-one to the R target second pixel points. For example, the R target features can be averaged or weighted averaged, and the feature of the first pixel point can be updated to the averaged result or weighted averaged result.

[0099] In the embodiments of the present disclosure, based on the similarity, the target second pixel point can be reasonably selected from the various second pixel points in the second feature map. The point information of the target second pixel point can provide an effective reference for updating the features of the first pixel point. On this basis, based on the pixel-level feature interaction between the first feature map and the second feature map, a third feature map that is more accurate and effective than the first feature map can be effectively obtained. The third feature map is used to update the first query vector, which can better guarantee the update effect of the first query vector, thereby facilitating the accuracy and reliability of the first detection result.

[0100] Figure 7 It is a flowchart of a method for selecting a target second pixel point that satisfies a preset feature similarity condition with a first pixel point from each second pixel point in a second feature map based on similarity, provided by some exemplary embodiments of the present disclosure. Figure 7 The method shown may include step 710 , step 720 , step 730 , and step 740 .

[0101] Optionally, Figure 7 In the method shown, the second feature map may include a first scale sub-feature map and a second scale sub-feature map. Figure 5 As shown, the second feature map may include sub-feature maps of three scales, namely, sub-feature map 510, sub-feature map 520 and sub-feature map 530. Then, one of the sub-feature map 510, sub-feature map 520 and sub-feature map 530 can be used as a first-scale sub-feature map, and the other two of the sub-feature map 510, sub-feature map 520 and sub-feature map 530 can be used as a second-scale sub-feature map.

[0102] Step 710: Based on the similarity, select a target second pixel point that satisfies a preset feature similarity condition with the first pixel point from each second pixel point in the first-scale sub-feature map.

[0103] Optionally, the preset feature similarity condition can refer to the relevant introduction to step 410 above, which will not be repeated here.

[0104] Optionally, there may be a total of M1 second pixel points in the first scale sub-feature map, and the similarities between the features of the first pixel point and the features of the M1 second pixel points may be obtained from the similarity set to obtain M1 similarities. The M1 similarities may be arranged in order from large to small, and the similarity with the highest ranking may be selected therefrom, and the second pixel point corresponding to the selected similarity may be used as the target second pixel point in the first scale sub-feature map that satisfies the preset feature similarity condition with the first pixel point.

[0105] Step 720: Determine the size ratio relationship between the second scale sub-feature map and the first scale sub-feature map.

[0106] Optionally, Figure 5 The sub-feature graph 520 in the figure may be obtained by scaling the sub-feature graph 510, and the sub-feature graph 530 may be obtained by scaling the sub-feature graph 520. Then, for the second-scale sub-feature graph and the first-scale sub-feature graph, a first ratio of the width of the second-scale sub-feature graph to the width of the first-scale sub-feature graph may be determined, and the first ratio may be used to characterize the size ratio relationship between the second-scale sub-feature graph and the first-scale sub-feature graph. Alternatively, a second ratio of the height of the second-scale sub-feature graph to the height of the first-scale sub-feature graph may be determined, and the second ratio may be used to characterize the size ratio relationship between the second-scale sub-feature graph and the first-scale sub-feature graph. Alternatively, a third ratio of the width of the first-scale sub-feature graph to the width of the second-scale sub-feature graph may be determined, and the third ratio may be used to characterize the size ratio relationship between the second-scale sub-feature graph and the first-scale sub-feature graph.

[0107] Step 730 : Based on the size ratio relationship, the coordinates of the second pixel point of the target in the first scale sub-feature map are mapped to the second scale sub-feature map to obtain the mapping coordinates on the second scale sub-feature map.

[0108] Optionally, the coordinates of the target second pixel point in the first scale sub-feature map can be determined, and the determined coordinates can be mapped to the second scale sub-feature map. The obtained mapping coordinates can satisfy: the relative distribution of the position with the mapping coordinates in the second scale sub-feature map is called the first relative distribution, and the relative distribution of the position where the target second pixel point in the first scale sub-feature map is located in the first scale sub-feature map is called the second relative distribution, then the first relative distribution and the second relative distribution are consistent.

[0109] In one example, the coordinates of the target second pixel point in the first scale sub-feature map are expressed as (x1, y1). The first ratio in the above text is used to characterize the size ratio relationship between the second scale sub-feature map and the first scale sub-feature map, and the first ratio is expressed as k1, and the mapping coordinates on the second scale sub-feature map are expressed as (x2, y2), then: x2 = k1*x1, y2 = k1*y1.

[0110] Step 740: Based on the mapping coordinates, select another target second pixel point that satisfies a preset feature similarity condition with the first pixel point from each second pixel point in the second-scale sub-feature map.

[0111] Optionally, a second pixel point with mapping coordinates can be selected from the second scale sub-feature map, and the selected second pixel point can be used as another target second pixel point in the second scale sub-feature map that satisfies a preset feature similarity condition with the first pixel point. The other target second pixel point is a different target second pixel point (located in a different sub-feature map) from the target second pixel point selected in step 710.

[0112] In the embodiment of the present disclosure, when the second feature map belongs to a multi-scale feature map, the target second pixel point can be first determined in a sub-feature map of a certain scale, and then the second pixel points of other targets can be determined in sub-feature maps of other scales by coordinate mapping between different sub-feature maps. In this way, the similarity set does not need to include the similarity between the first pixel point and each second pixel point in the sub-feature map of the other scale, so the workload of determining the similarity is reduced, which is conducive to reducing redundant calculations and improving target detection efficiency.

[0113] Figure 8 It is a flowchart of a method for detecting a target object provided by some other exemplary embodiments of the present disclosure. Figure 8 The method shown may include step 810 and step 820. Optionally, step 820 may be an optional implementation of step 150 of the present disclosure.

[0114] Step 810: Based on the second feature graph, using the first information interaction network, update the first query vector to obtain a fourth query vector.

[0115] Optionally, the second feature map and the first query vector can be used as input data of the first information interaction network, and the first information interaction network can perform operations based on the input data to apply the attention mechanism to update the first query vector, thereby obtaining a fourth query vector that is more accurate and effective than the first query vector. For example, the fourth query vector can more accurately and effectively reflect the relevant information of the object to be detected. The number of fourth query vectors can also be V.

[0116] In some optional embodiments of the present disclosure, the first information interaction network may include a first self-attention network and a first cross-attention network; wherein the first self-attention network may be a network based on a self-attention mechanism, and the first cross-attention network may be a network based on a cross-attention mechanism. Fig. 9 As shown, step 810 may include step 910 and step 920 .

[0117] Step 910: Use the first self-attention network to update the first query vector to obtain an intermediate query vector.

[0118] Optionally, based on V first query vectors, Figure 2Multiple Sources in<Key,Value> In addition, each of the V first query vectors can be used as a query, so that the corresponding V attention values ​​can be obtained, and each of the V attention values ​​can be used as an intermediate query vector, so that there can be V intermediate query vectors.

[0119] Step 920: Based on the second feature map, the first cross-attention network is used to update the intermediate query vector to obtain a fourth query vector.

[0120] Optionally, based on the second feature map, Figure 2 Multiple Sources in<Key,Value> In addition, each of the V intermediate query vectors can be used as a query, so that the corresponding V attention values ​​can be obtained, and each of the V attention values ​​can be used as a fourth query vector, so that there can be V fourth query vectors.

[0121] In the above implementation, the first query vector can be updated through the self-attention mechanism. Since each query vector can carry relevant information of an object to be detected, an object to be detected can be used as an instance, which is equivalent to obtaining an intermediate query vector that is more accurate and effective than the first query vector through instance-level information interaction. Afterwards, the second feature map can be interacted with the intermediate query vector through the cross-attention mechanism to learn useful information from the second feature map for updating the intermediate query vector, thereby obtaining a fourth query vector that is more accurate and effective than the intermediate query vector. Therefore, through the use of the self-attention mechanism and the cross-attention mechanism, the first query vector can be effectively optimized, thereby obtaining a reasonable and reliable fourth query vector.

[0122] In some embodiments, the first information interaction network may also only include the first cross-attention network. Then, the first query vector may be updated directly based on the second feature map using the first cross-attention network to obtain the fourth query vector.

[0123] Step 820: Generate a first detection result of the object to be detected based on the third query vector, the fourth query vector and the prediction network.

[0124] Optionally, the V third query vectors and the V fourth query vectors can be used as input data of the prediction network, and the prediction network can decode the input data to obtain a first detection result. The first detection result can be represented by a second image with 2V two-dimensional detection boxes.

[0125] In the embodiments of the present disclosure, based on the first information interaction network, information interaction between the second feature map and the first query vector can be effectively realized to obtain a fourth query vector that is more accurate and effective than the first query vector. Combined with the third query vector and the prediction network, the first detection result can be reasonably determined, which is conducive to ensuring the accuracy, reliability and completeness of the first detection result.

[0126] Fig.10 It is a flowchart of a method for detecting a target object provided by some further exemplary embodiments of the present disclosure. Fig.10 The method shown may include step 1010, step 1020, step 1030 and step 1040. Optionally, Fig.10 The method shown can be applied to the case where the first image and the second image are captured at different times.

[0127] Step 1010: Based on the second feature graph and the first information interaction network, update the first query vector to obtain a fourth query vector.

[0128] Optionally, the specific implementation of step 1010 may refer to the above description of step 810 and will not be repeated here.

[0129] Step 1020: Update the fourth query vector based on the first feature map and the second information interaction network to obtain a fifth query vector.

[0130] Optionally, the second information interaction network may include a second self-attention network and a second cross-attention network; wherein the second self-attention network may be a network based on a self-attention mechanism, and the second cross-attention network may be a network based on a cross-attention mechanism. Fig. 9 Similar to the illustrated implementation, in step 1020, the fourth query vector can be updated by using the second self-attention network to obtain an intermediate query vector, and then based on the first feature map, the second cross-attention network can be used to update the intermediate query vector to obtain a fifth query vector.

[0131] Step 1030: Generate a second detection result of the object to be detected based on the fifth query vector and the prediction network.

[0132] Optionally, the specific implementation of step 1030 can refer to the relevant introduction of step 150 above, which will not be repeated here. The second detection result of the object to be detected can be used as the target detection result of the first image. The second detection result can be used to characterize which objects to be detected exist in the first image, and / or the position, size, etc. of these objects to be detected in the first image.

[0133] Step 1040: Determine, based on the first detection result and the second detection result, newly added information and / or missing information of the object to be detected between the second image and the first image.

[0134] In one example, the second detection result is used to characterize that there are 5 objects to be detected in the first image, namely, object 1 to be detected, object 2 to be detected, object 3 to be detected, object 4 to be detected, and object 5 to be detected. The first detection result is used to characterize that there are 4 objects to be detected in the second image, namely, object 1 to be detected, object 2 to be detected, object 6 to be detected, and object 7 to be detected. Obviously, compared with the first image, 2 new objects to be detected appear in the second image, namely, object 6 to be detected and object 7 to be detected, and, compared with the first image, 3 objects to be detected disappear in the second image, namely, object 3 to be detected, object 4 to be detected, and object 5 to be detected. Then, the newly added information of the objects to be detected determined in step 1040 can be used to indicate that objects 6 to be detected and object 7 to be detected are newly added, and the disappearance information of the objects to be detected determined in step 1040 can be used to indicate that objects 3 to be detected, object 4 to be detected, and object 5 to be detected disappear.

[0135] In the embodiments of the present disclosure, based on the second feature map and the first information interaction network, a fourth query vector that is more accurate and effective than the first query vector can be obtained through information interaction. Based on the first feature map and the second information interaction network, a fifth query vector that is more accurate and effective than the fourth query vector can be obtained through information interaction, which can provide an effective reference for the generation of the second detection result of the object to be detected. On this basis, combined with the first detection result, the detection of the object to be detected that has changed (i.e., appeared or disappeared) between images can be effectively realized, then the target detection model has a good detection capability for the object to be detected that has changed. For example, the target detection module can accurately and stably detect the object to be detected that has changed.

[0136] Fig.11 It is a flowchart of a method for updating a second query vector to obtain a third query vector based on a second feature map and a second information interaction network provided by some exemplary embodiments of the present disclosure. Fig.11 The method shown may include step 1110 , step 1120 , and step 1130 .

[0137] Step 1110, determining the similarity between the feature of the first pixel in the first feature map and the feature of the second pixel in the second feature map.

[0138] Step 1120: Based on the first feature map and the similarity, update the feature of the second pixel in the second feature map to obtain a fourth feature map.

[0139] Step 1130: Based on the fourth feature map and the second information interaction network, update the second query vector to obtain a third query vector.

[0140] Optionally, the specific implementation of steps 1110 to 1130 may refer to the above description of steps 310 to 330, which will not be repeated here.

[0141] In an embodiment of the present disclosure, after determining the similarity between the feature of the first pixel in the first feature map and the feature of the second pixel in the second feature map, the feature of the second pixel in the second feature map can be updated based on the first feature map and the similarity, which is equivalent to performing pixel-level feature interaction on the first feature map and the second feature map, thereby obtaining a fourth feature map that is more accurate and effective than the second feature map. For example, the features in the fourth feature map can more accurately and effectively represent the relevant information of the object to be detected. In addition, based on the second information interaction network, the information interaction between the fourth feature map and the second query vector can be effectively realized to obtain a third query vector that is more accurate and effective than the second query vector. The first detection result is determined based on the third query vector, which can better ensure the accuracy and reliability of the first detection result.

[0142] In some embodiments, the process of obtaining the third query vector may not use similarity. For example, the second information interaction network may include a second self-attention network and a second cross-attention network. Then, the second query vector may be updated using the second self-attention network to obtain an intermediate query vector, and then the intermediate query vector may be updated based on the second cross-attention network to obtain the third query vector.

[0143] In some optional examples, the same scene can be photographed by the same camera at different shooting times to obtain a first image and a second image. A second feature map can be obtained by extracting features from the second image. The second feature map can be a multi-scale feature map, for example, including Figure 5 The sub-feature graph 510, the sub-feature graph 520 and the sub-feature graph 530 in FIG. Similarly, by performing feature extraction on the first image, a first feature graph can be obtained. The first feature graph can belong to a multi-scale feature graph, for example, including Figure 5 Sub-feature graph 540, sub-feature graph 550 and sub-feature graph 560 in.

[0144] Optionally, the first feature map can be represented as X_l, and the second feature map can be represented as X_r. Based on the self-attention mechanism, X_l can be updated to X_l', and X_r can be updated to X_r'. Based on the cross-attention mechanism, X_l' can be updated to X_l'', and X_r'' can be updated to X_r''. The process of obtaining X_l''' from X_l and obtaining X_r''' from X_r can be considered as one round of updating of the feature map. There can be a total of 6 rounds of updating of the feature map, and the result of X_l after 6 rounds of updating can be represented as X_l'''', and the result of X_r after 6 rounds of updating can be represented as X_r''''. Based on X_l'''' and X_r'''', the target detection results of the first image and the second image can be determined respectively.

[0145] Figure 5 In the figure, the circular pattern is equivalent to the reference point mentioned above, the five-pointed star pattern is equivalent to the sampling point mentioned above, the arrowed line segment from X_l to X_l' and the arrowed line segment from X_r to X_r' represent the self-attention mechanism, and the arrowed line segment from X_l' to X_r'' and the arrowed line segment from X_r' to X_l'' represent the cross-attention mechanism.

[0146] It should be noted that in the process of determining the target detection results, based on the self-attention mechanism, information interaction within the graph can be achieved, and based on the cross-attention mechanism, information interaction between graphs can be achieved, thereby better ensuring the accuracy and reliability of the target detection results.

[0147] In some optional examples, the third feature map may belong to a multi-scale feature map, for example including Fig.12 , 1210, 1220, and 1230. Similarly, the fourth feature map may belong to a multi-scale feature map, for example, including Fig.12 Sub-feature graph 1240, sub-feature graph 1250 and sub-feature graph 1260 in.

[0148] Optionally, the first query vector can be obtained through an initialization operation. Based on the third feature map and the fourth feature map, the first query vector can be updated several times through the self-attention mechanism and the cross-attention mechanism to obtain the third query vector and the fourth query vector mentioned above, and then combined with the prediction network to generate the target detection result of the second image. The target detection result of the first image can be obtained in a similar way, which will not be repeated here.

[0149] Fig.12In the figure, the circular pattern is equivalent to the reference point mentioned above, the five-pointed star pattern is equivalent to the sampling point mentioned above, the triangle pattern represents the query vector, the arrowed line segment from the third feature map to the query vector and the arrowed line segment from the fourth feature map to the query vector represent the cross attention mechanism, the solid line with an arrow from the query vector to the query vector represents the self-attention mechanism, and the dashed line with an arrow from the query vector to the query vector represents the identity mapping. As introduced above, Figure 5 In the illustrated embodiment, the feature map may be updated in 6 rounds, that is, the feature map may be updated iteratively. Similarly, Fig.12 In the illustrated embodiment, the query vector may also be updated iteratively.

[0150] In summary, by adopting the embodiments of the present disclosure, it is possible to efficiently and reliably realize intra-image information interaction and inter-image information interaction, so as to optimize the representation of the query vector and obtain a query vector suitable for mutual decoding of two images (for example, a first image and a second image). On this basis, a complete, accurate and reliable target detection result can be obtained.

[0151] Exemplary Devices

[0152] Fig.13 It is a schematic diagram of the structure of a target object detection device provided by some exemplary embodiments of the present disclosure. Fig.13 The apparatus shown may include:

[0153] An acquisition module 1310 is used to acquire a first image and a second image obtained by photographing the same scene; wherein the first image and the second image are photographed at different times and / or at different viewing angles;

[0154] The feature extraction module 1320 is used to extract features from the first image acquired by the acquisition module 1310 to obtain a first feature map, and to extract features from the second image acquired by the acquisition module 1310 to obtain a second feature map;

[0155] A first updating module 1330 is used to update the first query vector of the object to be detected based on the first feature map and the second feature map obtained by the feature extraction module 1320 and the first information interaction network to obtain a second query vector;

[0156] A second updating module 1340 is configured to update the second query vector obtained by the first updating module 1330 based on the second feature graph and the second information interaction network obtained by the feature extraction module 1320 to obtain a third query vector;

[0157] The first generating module 1350 is used to generate a first detection result of the object to be detected based on the third query vector and the prediction network obtained by the second updating module 1340.

[0158] In some optional examples, such as Figure 14-1 As shown, the device provided by the embodiment of the present disclosure also includes:

[0159] A third updating module 1410 is configured to update the first query vector based on the second feature map obtained by the feature extraction module 1320 and to obtain a fourth query vector by using the first information interaction network;

[0160] The first generating module 1350 is used to generate a first detection result of the object to be detected based on the third query vector obtained by the second updating module 1340, the fourth query vector obtained by the third updating module 1410 and the prediction network.

[0161] In some optional examples, such as Figure 14-2 As shown, the device provided by the embodiment of the present disclosure also includes:

[0162] A fourth updating module 1420 is configured to update the first query vector based on the second feature graph obtained by the feature extraction module 1320 and the first information interaction network to obtain a fourth query vector;

[0163] A fifth updating module 1430 is configured to update the fourth query vector obtained by the fourth updating module 1420 based on the first feature graph and the second information interaction network obtained by the feature extraction module 1320 to obtain a fifth query vector;

[0164] A second generating module 1440, configured to generate a second detection result of the object to be detected based on the fifth query vector and the prediction network obtained by the fifth updating module 1430;

[0165] The determination module 1450 is used to determine the newly added information and / or disappeared information of the object to be detected between the second image and the first image based on the first detection result generated by the first generation module 1350 and the second detection result generated by the second generation module 1440.

[0166] In some optional examples, such as Fig.15 As shown, the first update module 1330 includes:

[0167] A first determination submodule 1510 is used to determine the similarity between the feature of the first pixel point in the first feature map obtained by the feature extraction module 1320 and the feature of the second pixel point in the second feature map obtained by the feature extraction module 1320;

[0168] A first updating submodule 1520 is used to update the feature of the first pixel in the first feature map obtained by the feature extraction module 1320 based on the second feature map obtained by the feature extraction module 1320 and the similarity determined by the first determining submodule 1510 to obtain a third feature map;

[0169] The second updating submodule 1530 is used to update the first query vector based on the third feature map obtained by the first updating submodule 1520 and the first information interaction network to obtain a second query vector.

[0170] In some optional examples, such as Fig.16 As shown, the first updating submodule 1520 includes:

[0171] A selection unit 1610 is used to select a target second pixel point that satisfies a preset feature similarity condition with the first pixel point from each second pixel point in the second feature map obtained by the feature extraction module 1320 based on the similarity determined by the first determination submodule 1510;

[0172] The updating unit 1620 is used to update the characteristics of the first pixel point based on the point information of the target second pixel point selected by the selecting unit 1610.

[0173] In some optional examples, such as Fig.17 As shown, the updating unit 1620 includes:

[0174] A first determining subunit 1710 is used to determine the coordinates and weight of each second pixel point in the second feature map obtained by the feature extraction module 1320, based on the point information of the target second pixel point selected by the selection unit 1610, of a plurality of second pixel points that meet a preset sampling condition;

[0175] The updating subunit 1720 is used to update the characteristics of the first pixel point based on the coordinates and weights of each second pixel point that meets the preset sampling condition and is determined by the first determining subunit 1710.

[0176] In some optional examples, the second feature map includes a first-scale sub-feature map and a second-scale sub-feature map;

[0177] like Fig.18 As shown, the selection unit 1610 includes:

[0178] A first selection subunit 1810 is used to select, from each second pixel point in the first scale sub-feature map, a target second pixel point that satisfies a preset feature similarity condition with the first pixel point based on the similarity determined by the first determination submodule 1510;

[0179] The second determining subunit 1820 is used to determine the size ratio relationship between the second scale sub-feature map and the first scale sub-feature map;

[0180] A mapping subunit 1830 is used to map the coordinates of the second pixel point of the target in the first scale sub-feature map to the second scale sub-feature map based on the size ratio relationship determined by the second determining subunit 1820, so as to obtain the mapping coordinates on the second scale sub-feature map;

[0181] The second selection subunit 1840 is used to select another target second pixel point that satisfies a preset feature similarity condition with the first pixel point from each second pixel point in the second scale sub-feature map based on the mapping coordinates obtained by the mapping subunit 1830.

[0182] In the device of the present disclosure, the various optional embodiments, optional implementation methods and optional examples disclosed above can be flexibly selected and combined as needed to achieve corresponding functions and effects, and the present disclosure does not list them one by one.

[0183] Exemplary Electronic Devices

[0184] Fig.19 A block diagram of an electronic device according to an embodiment of the present disclosure is illustrated. The electronic device 1900 includes one or more processors 1910 and a memory 1920 .

[0185] The processor 1910 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 1900 to perform desired functions.

[0186] The memory 1920 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 1910 may execute one or more computer program instructions to implement the methods of the various embodiments of the present disclosure described above and / or other desired functions.

[0187] In one example, the electronic device 1900 may further include: an input device 1930 and an output device 1940 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0188] The input device 1930 may also include, for example, a keyboard, a mouse, etc.

[0189] The output device 1940 can output various information to the outside, and may include, for example, a display, a speaker, a printer, a communication network and its connected remote output device, etc.

[0190] Of course, to simplify, Fig.19Only some of the components related to the present disclosure in the electronic device 1900 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application scenarios, the electronic device 1900 may also include any other appropriate components.

[0191] Exemplary computer program products and computer-readable storage media

[0192] In addition to the above-mentioned methods and devices, an embodiment of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the method according to various embodiments of the present disclosure described in the above-mentioned "Exemplary Method" section of this specification.

[0193] The computer program product may be written in any combination of one or more programming languages ​​to write program code for performing the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0194] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, enables the processor to execute the steps of the method according to various embodiments of the present disclosure described in the above “Exemplary Method” section of this specification.

[0195] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0196] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations. It cannot be considered that these advantages, strengths, effects, etc. must be possessed by each embodiment of the present disclosure. The specific details disclosed above are only for the purpose of illustration and ease of understanding, not limitation. The above details do not limit the present disclosure to being implemented by adopting the above specific details.

[0197] Those skilled in the art may make various changes and modifications to the present disclosure without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the present disclosure claims and their equivalents, the present disclosure is also intended to include these modifications and variations.

Claims

1. A method for detecting a target object, comprising: Acquire a first image and a second image obtained by photographing the same scene; wherein the first image and the second image are photographed at different times and / or at different viewing angles; Performing feature extraction on the first image to obtain a first feature map, and performing feature extraction on the second image to obtain a second feature map; Based on the first feature map, the second feature map and the first information interaction network, updating a first query vector of the object to be detected to obtain a second query vector; Based on the second feature graph and the second information interaction network, updating the second query vector to obtain a third query vector; Based on the third query vector and the prediction network, a first detection result of the object to be detected is generated.

2. The method according to claim 1, wherein: The method further comprises: Based on the second feature graph, using the first information interaction network, updating the first query vector to obtain a fourth query vector; The step of generating a first detection result of the object to be detected based on the third query vector and the prediction network includes: A first detection result of the object to be detected is generated based on the third query vector, the fourth query vector and the prediction network.

3. The method according to claim 1, wherein: The method further comprises: Based on the second feature graph and the first information interaction network, updating the first query vector to obtain a fourth query vector; Based on the first feature graph and the second information interaction network, updating the fourth query vector to obtain a fifth query vector; generating a second detection result of the object to be detected based on the fifth query vector and the prediction network; Based on the first detection result and the second detection result, newly added information and / or disappeared information of the object to be detected between the second image and the first image is determined.

4. The method according to claim 1, wherein: The updating of the first query vector of the object to be detected based on the first feature map, the second feature map and the first information interaction network to obtain the second query vector includes: Determine a similarity between a feature of a first pixel in the first feature map and a feature of a second pixel in the second feature map; Based on the second feature map and the similarity, updating the feature of the first pixel in the first feature map to obtain the third feature map; Based on the third feature graph and the first information interaction network, the first query vector is updated to obtain the second query vector.

5. The method according to claim 4, wherein: The updating the feature of the first pixel in the first feature map based on the second feature map and the similarity includes: Based on the similarity, selecting a target second pixel point that satisfies a preset feature similarity condition with the first pixel point from each second pixel point in the second feature map; Based on the point information of the target second pixel point, the feature of the first pixel point is updated.

6. The method according to claim 5, wherein: The updating of the feature of the first pixel point based on the point information of the second pixel point of the target includes: Based on the point information of the target second pixel point, determine the coordinates and weight of each second pixel point in the second feature map, among the plurality of second pixel points that meet the preset sampling condition; Based on the coordinates and weight of each second pixel point that meets the preset sampling condition, the features of the first pixel point are updated.

7. The method according to claim 5, wherein: The second feature map includes a first scale sub-feature map and a second scale sub-feature map; The selecting, based on the similarity, from each second pixel point in the second feature map, a target second pixel point that satisfies a preset feature similarity condition with the first pixel point comprises: Based on the similarity, selecting a target second pixel point that satisfies a preset feature similarity condition with the first pixel point from each second pixel point in the first-scale sub-feature map; Determining a size ratio relationship between the second-scale sub-feature map and the first-scale sub-feature map; Based on the size ratio relationship, mapping the coordinates of the second pixel point of the target in the first scale sub-feature map to the second scale sub-feature map to obtain the mapping coordinates on the second scale sub-feature map; Based on the mapping coordinates, another target second pixel point that satisfies a preset feature similarity condition with the first pixel point is selected from each second pixel point in the second-scale sub-feature map.

8. A device for detecting a target object, comprising: An acquisition module, used to acquire a first image and a second image obtained by photographing the same scene; wherein the first image and the second image are photographed at different times and / or at different viewing angles; a feature extraction module, configured to extract features from the first image acquired by the acquisition module to obtain a first feature map, and to extract features from the second image acquired by the acquisition module to obtain a second feature map; A first updating module, configured to update a first query vector of the object to be detected based on the first feature map and the second feature map obtained by the feature extraction module and a first information interaction network to obtain a second query vector; A second updating module, configured to update the second query vector obtained by the first updating module based on the second feature graph and the second information interaction network obtained by the feature extraction module to obtain a third query vector; The first generating module is used to generate a first detection result of the object to be detected based on the third query vector and the prediction network obtained by the second updating module.

9. A computer-readable storage medium storing a computer program, wherein the computer program is used to execute the target object detection method according to any one of claims 1 to 7.

10. An electronic device, comprising: processor; a memory for storing instructions executable by the processor; The processor is used to read the executable instructions from the memory and execute the instructions to implement the target object detection method described in any one of claims 1-7.

Images