Target detection methods, devices and electronic equipment

Abstract

The present disclosure provides a target detection method, device and electronic device, which are related to artificial intelligence technology, especially to computer vision, deep learning and other technical fields. The specific implementation scheme is: obtaining a multi-scale feature map of a multi-view image; determining the three-dimensional coordinate information of each scale feature map in space, wherein the three-dimensional coordinate information of each scale feature map includes the three-dimensional coordinates of multiple points corresponding to each pixel point in the scale feature map in space; encoding each scale feature map and the three-dimensional coordinate information of the scale feature map, and determining the three-dimensional position perception feature of the multi-scale feature map according to the encoding result; decoding based on the three-dimensional position perception feature of the multi-scale feature map, and obtaining the object category and position information of the object in the multi-view image. It is helpful to improve the accuracy of 3D target detection.

Description

Technical Field

[0001] The present disclosure relates to technical fields such as computer vision and deep learning in the field of artificial intelligence, and can be applied to scenarios such as autonomous driving, virtual reality, industrial automation, and security, and in particular to a target detection method, device, and electronic device. Background Art

[0002] In many scenarios, three-dimensional (3D) target detection is required. For example, in autonomous driving scenarios, 3D target detection is needed to assist the vehicle in accurately identifying and locating targets such as pedestrians, vehicles, and obstacles on the road, and then provide the detection results to the autonomous driving system for decision-making and control. For example, in industrial automation scenarios, industrial robots need to be accurately positioned and controlled through 3D target detection; products on the production line need to be quality-checked and classified through 3D detection. In the field of security monitoring, 3D target detection is needed to monitor information such as the flow of people and vehicles in public places, airports, and railway stations in order to provide real-time security alerts and warnings. In virtual reality scenarios, 3D target detection is needed to identify and detect the positions of virtual objects in virtual reality scenes.

[0003] You can use multi-view Figure 2 3D target detection is performed on objects in the environment using multi-dimensional images. Figure 2 When performing 3D object detection on multi-dimensional images, the accuracy and efficiency of 3D object detection still need to be improved. Summary of the invention

[0004] The present disclosure provides a target detection method, device and electronic device.

[0005] According to a first aspect of the present disclosure, a target detection method is provided, the method comprising: obtaining a multi-scale feature map of a multi-view image; determining three-dimensional coordinate information of each scale feature map in space, wherein the three-dimensional coordinate information of each scale feature map includes the three-dimensional coordinates of multiple points corresponding to each pixel point in the scale feature map in space; encoding each scale feature map and the three-dimensional coordinate information of the scale feature map, and determining a three-dimensional position perception feature of the multi-scale feature map according to the encoding result; decoding based on the three-dimensional position perception feature of the multi-scale feature map to obtain an object category and position information of the object in the multi-view image.

[0006] According to a second aspect of the present disclosure, a target detection device is provided, the device comprising: an acquisition unit, which acquires a multi-scale feature map of a multi-view image; a determination unit, which is used to determine the three-dimensional coordinate information of each scale feature map in space, wherein the three-dimensional coordinate information of each scale feature map includes the three-dimensional coordinates of multiple points corresponding to each pixel point in the scale feature map in space; an encoding unit, which is used to encode each scale feature map and the three-dimensional coordinate information of the scale feature map, and determine the three-dimensional position perception feature of the multi-scale feature map according to the encoding result; a decoding unit, which is used to decode based on the three-dimensional position perception feature of the multi-scale feature map to obtain the object category and position information of the object in the multi-view image.

[0007] According to a third aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method provided in the first aspect.

[0008] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method provided in the first aspect.

[0009] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising: a computer program, wherein the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and the at least one processor executes the computer program so that the electronic device executes the method described in the first aspect.

[0010] According to the scheme disclosed in the present invention, a multi-scale feature map of a multi-view image is obtained; the three-dimensional coordinate information of each scale feature map in space is determined, wherein the three-dimensional coordinate information of each scale feature map includes the three-dimensional coordinates of multiple points corresponding to each pixel point in the scale feature map in space; each scale feature map and the three-dimensional coordinate information of the scale feature map are encoded, and the three-dimensional position perception features of the multi-scale feature map are determined according to the encoding result; the three-dimensional position perception features of the multi-scale feature map are decoded to obtain the object category and position information of the object in the multi-view image. By constructing the 3D position coordinate information corresponding to different scale features, the 3D position encoding is extended to the multi-scale representation to realize the implicit 2D to 3D position relationship learning, which can provide more image features for object query and help improve the accuracy and efficiency of 3D target detection.

[0011] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0013] Figure 1 is a scene graph that can implement the embodiments of the present disclosure;

[0014] Figure 2 is a schematic diagram according to a first embodiment of the present disclosure;

[0015] Figure 3 is a schematic diagram according to a second embodiment of the present disclosure;

[0016] Figure 4 is a schematic diagram according to a third embodiment of the present disclosure;

[0017] Figure 5 It is a schematic diagram of a network structure for realizing a target detection method;

[0018] Figure 6 is a schematic diagram according to a fourth embodiment of the present disclosure;

[0019] Figure 7 is a schematic block diagram of an example electronic device that can be used to implement embodiments of the present disclosure. DETAILED DESCRIPTION

[0020] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0021] Please refer to Figure 1 , which shows a schematic diagram of the network structure for implementing 3D object detection. Figure 1 As shown, multiple images of the environment with a 360° view can be obtained, for example, 6 images 11, 12, 13, 14, 15, and 16 can be viewed around the environment, and then these 6 images are input into the backbone network to extract the two-dimensional feature maps of the multiple views. These feature maps are then input into the feature pyramid network (FPN) for enhancement, and the enhanced feature maps are output. Each pixel in the feature map represents a feature.

[0022] The 2D feature maps of multiple views are input into the 3D coordinate generator, where the camera cone space shared by all views is discretized into a 3D grid, and the 3D coordinates of multiple points on the ray starting from each pixel point in the 2D feature map are obtained. Then, multiple 3D coordinates corresponding to each pixel in the 2D feature map are obtained.

[0023] The features in the feature map and the multiple 3D coordinates corresponding to each pixel in the feature map are input into the encoder to generate 3D position-aware features.

[0024] The 3D position-aware features are input to the decoder, which decodes the 3D position-aware features and the object query vector to obtain the object category and object position of the objects in the multi-view. Each object query vector can be a representation of an object in the detection task.

[0025] The object query vector may include query position (query_pos) vectors corresponding to the three-dimensional space. In the decoder, all object query vectors are processed by self-attention mechanism. After being processed by the self-attention mechanism, they are processed by cross-attention mechanism with the input 3D position perception features. When the cross-attention mechanism is processed, the object query vector of the 3D reference point needs to interact with the 3D position perception features of the corresponding point in the 2D image to update the object query vector corresponding to the 3D reference point, thereby integrating the 3D position perception features of the two-dimensional image into the object query vector. The object query vector processed by the cross-attention mechanism is input into the feed-forward network (FFN) and the prediction network, and the prediction network outputs the object category and position of the object.

[0026] exist Figure 1 In the implementation scheme shown, multiple 3D coordinates of each pixel of the feature map are fused and encoded with the image features of the pixel into 3D position-aware features, thereby achieving implicit 2D to 3D position embedding. When performing target detection on the environment image according to the above scheme, less computational effort and better target detection results can be achieved.

[0027] but Figure 1 In the solution shown, since there may be objects of different sizes in the scene, use Figure 1 The scheme shown cannot accurately detect objects of different sizes. In addition, although a 2D to 3D implicit embedding scheme is used when performing the cross-attention mechanism, the image features provided by the above scheme to the object query vector are not sufficient. Therefore, the accuracy and precision of the object detection results obtained by the above scheme need to be improved.

[0028] The present disclosure provides a target detection method, device and electronic device, which are applied to technical fields such as computer vision and deep learning in the field of artificial intelligence to achieve the purpose of improving the accuracy and precision of target detection results.

[0029] Please refer to Figure 2 , Figure 2 is a schematic diagram of the first embodiment of the present disclosure, such as Figure 2 As shown, the target detection method provided in this embodiment includes the following steps:

[0030] S201. Obtain a multi-scale feature map of a multi-view image.

[0031] The execution subject of this embodiment can be a terminal device or a server. In this embodiment, the execution subject is described as a terminal device.

[0032] The multi-view images include multiple images taken at different shooting angles, and each image may be a two-dimensional (2D) image.

[0033] The multi-view images may be multiple images captured by the same image capture device at different viewing angles, or may be multiple images captured by different image capture devices at different viewing angles at the same time.

[0034] In one example, the above-mentioned obtaining a multi-scale feature map of a multi-view image may include obtaining a multi-scale feature map of each image in the multi-view image, and then splicing each scale feature map of each view image to obtain a multi-scale feature map.

[0035] In one example, the above multi-view images can be integrated into a scene image, and then multi-scale feature extraction is performed on the image data of the scene image. A feature map of an image is the result of a series of convolution kernels performing a convolution operation on the input image data of the image, and each pixel in the feature map represents some specific features.

[0036] The multi-scale feature map may include feature maps of different scales obtained by downsampling the multi-view image data using different sampling multiples. The downsampling multiples may include, for example, 4, 8, 16, 32 times, etc. For an image with an original size of W×H, after downsampling the image by the above sampling multiples, images of sizes (W / 4)×(H / 4), (W / 8)×(H / 8), (W / 16)×(H / 16), and (W / 32)×(H / 32) may be obtained. Each feature map obtained by each sampling multiple in the multi-scale feature map may correspond to a level of 2D features.

[0037] In one example, the image data of the W×H image can be input into a multi-layer convolution kernel, and each convolution kernel can downsample the image data input therein. For example, using an S×S convolution kernel to downsample the image data input therein can convert the image data within the S×S window input therein into one pixel. Where S is an integer greater than 1.

[0038] As the number of convolution kernel layers increases, the downsampling multiple also increases. According to which downsampling multiple feature map is required, it can be determined which layer of convolution kernel outputs the feature map of the downsampling multiple.

[0039] Schematically, the multi-scale feature map includes feature maps of scales such as 1 / 4, 1 / 8, and 1 / 16 of the original image size.

[0040] The scene to be detected may include objects of different sizes. As the depth of the neural network increases, the downsampling multiple increases, the receptive field increases, and the semantic information increases, which is suitable for detecting large objects, but small objects may be submerged. Therefore, when using image features obtained with higher sampling multiples for target detection, it is possible that the problem of not being able to accurately identify smaller objects may occur. Therefore, it is necessary to use multi-scale feature maps to construct image features of objects of different sizes in order to achieve accurate detection results when performing target detection in scenes including objects of different sizes.

[0041] S202: Determine three-dimensional coordinate information of each scale feature map in space, wherein the three-dimensional coordinate information of each scale feature map includes three-dimensional coordinates of a plurality of points corresponding to each pixel point in the scale feature map in space.

[0042] The coordinates of the pixels in the feature map in the three-dimensional space can be determined in various ways.

[0043] In this embodiment, the image data of the multi-view image may include the camera parameters (including intrinsic parameters and extrinsic parameters) of the camera that captured the above image. The coordinates of the pixel points in the feature map in the three-dimensional space can be calculated based on the pixel points in the feature map and the camera parameter matrix of the camera.

[0044] As an implementation manner, the above step S202 may include: determining the three-dimensional coordinate information of each scale feature map in the viewing cone space corresponding to the multiple views.

[0045] In order to establish the connection between 2D images and 3D space, a camera cone space shared by multiple views is constructed, and the cone space is discretized into a 3D grid. In the above process, for any pixel point in each feature map, the ray starting from the pixel point can include multiple three-dimensional space coordinates. In this way, each pixel point can correspond to multiple three-dimensional coordinates in space, for example, D three-dimensional coordinates, where D is an integer greater than or equal to 2.

[0046] S203: Encode each scale feature map and the three-dimensional coordinate information of the scale feature map, and determine the three-dimensional position perception features of the multi-scale feature map according to the encoding result.

[0047] Assume that the multi-scale feature map (2D image feature) of the multi-view image is represented by the following formula:

[0048]

[0049] Where l is the level of the index 2 image feature (for example, the level formed by the feature maps of scales 1 / 4, 1 / 8, 1 / 16 of the original size, schematically, the 1 / 4 scale feature map is level 1, the 1 / 8 scale feature map is level 2, and the 1 / 16 scale feature map is level 3), N represents the number of views in the multi-view, and W l and H l are the width and height of the l-th level feature respectively. L is the maximum number of levels, and L is an integer greater than 1.

[0050] 3D Position Mapping of Multi-Scale Feature Maps It can be represented by the following formula:

[0051]

[0052] in It is a point in the image feature space in the 3D grid, where D represents the number of three-dimensional coordinates in the viewing cone space corresponding to each pixel point. is the 3D coordinate after inverse 3D projection. K represents the internal and external parameters of the camera. PE() is the encoding algorithm.

[0053] By constructing 3D position codes corresponding to feature maps of different scales, the 3D position codes are extended to multi-scale representations to achieve implicit 2D to 3D position relationship learning. This can provide more image features for the object query vector during decoding, which helps to improve the accuracy of 3D target detection.

[0054] S204 , decoding is performed based on the three-dimensional position-aware features of the multi-scale feature map to obtain object categories and position information of objects in the multi-view image.

[0055] The above step S204 includes: decoding based on the three-dimensional position perception features of the multi-scale feature map and a plurality of preset object query vectors, where the object query vector includes an encoding vector of a position in the three-dimensional space.

[0056] Since the above-mentioned three-dimensional position perception features include three-dimensional position information and image feature information, a more accurate target detection result can be obtained by decoding the three-dimensional position perception features corresponding to the above-mentioned multi-scale feature maps.

[0057] The object query vector may include query position (query_pos) vectors and type query (query) vectors corresponding to multiple three-dimensional spaces. For each object query vector, the query position corresponding to the object query vector may be mapped to the Bird's Eye View (BEV) space to obtain the three-dimensional reference point of the BEV space corresponding to the object query vector. The three-dimensional reference point corresponding to the object query vector is a learnable reference point.

[0058] The three-dimensional position-aware features corresponding to the multi-scale feature maps can interact with multiple object query vectors, and the object query vectors can be updated according to the interaction results. The object type and the object position can be predicted according to the updated object query vectors.

[0059] In this embodiment, a multi-scale feature map of a multi-view image is obtained; the three-dimensional coordinate information of each scale feature map in the viewing cone space is determined, wherein the three-dimensional coordinate information of each feature map includes the three-dimensional coordinates of multiple points in the viewing cone space for each pixel point in the feature map; each scale feature map and the three-dimensional coordinate information of the scale feature map are encoded to obtain three-dimensional position perception features corresponding to the multi-scale feature maps; based on the three-dimensional position perception features corresponding to the multi-scale feature maps, decoding is performed to obtain object categories and position information of objects in the multi-view images, and by constructing 3D position coordinate information corresponding to image features of different scales, the 3D position encoding is extended to multi-scale representation to realize implicit 2D to 3D position relationship learning, which can provide more image features for the object query vector and help improve the accuracy of 3D target detection of the object.

[0060] Please refer to Figure 3 , Figure 3 is a schematic diagram of the second embodiment of the present disclosure, such as Figure 3 As shown, the target detection method provided in this embodiment includes the following steps:

[0061] S301, obtaining a multi-scale feature map of a multi-view image.

[0062] S302: Determine three-dimensional coordinate information of each scale feature map in space, wherein the three-dimensional coordinate information of each scale feature map includes three-dimensional coordinates of multiple points corresponding to each pixel point in the scale feature map in space.

[0063] S303: Encode each scale feature map and the three-dimensional coordinate information of the scale feature map, and determine the three-dimensional position perception features of the multi-scale feature map according to the encoding result.

[0064] The specific implementation of the above steps S301 to S303 can refer to Figure 2 The relevant parts of the illustrated embodiment will not be described in detail here.

[0065] In this embodiment, the target detection method may be executed by a terminal device or a server that provides services to the terminal device.

[0066] S304. For each object query vector, bidirectional offset is performed according to a reference point of the object query vector in the three-dimensional space to obtain a plurality of two-dimensional offset points of the reference point in the image space; wherein the bidirectional offset includes performing a first offset on the reference point in the three-dimensional space, and performing a second offset on the three-dimensional offset point obtained by the first offset and the projection point of the reference point in the two-dimensional space, respectively.

[0067] In this embodiment, in order to further solve the problem of insufficient image features provided for the object query vector, bidirectional offset is performed to determine multiple two-dimensional offset points in the image for each reference point of the object query vector, and then the image features related to the above two-dimensional offset points are embedded into the object query vector through a cross-attention mechanism.

[0068] The above-mentioned bidirectional offset may include, in three-dimensional space, performing a first offset in three-dimensional space on a reference point corresponding to the object query vector, and performing a second offset in two-dimensional space on the three-dimensional offset point obtained by the first offset and the projection point of the reference point in two-dimensional space. Through the above-mentioned bidirectional offset, a larger number of two-dimensional offset points associated with the reference point of the object query vector may be obtained. By integrating the image features corresponding to the above-mentioned multiple two-dimensional offset points into the object query vector, the object query vector may have more image information.

[0069] Specifically, the above step S304 may include the following steps:

[0070] First, a reference point of the object query vector in the three-dimensional space is first offset in the three-dimensional space to determine at least one three-dimensional offset point.

[0071] For each object query vector, at least one three-dimensional offset point is determined according to a reference point of the object query vector in the three-dimensional space.

[0072] At least one three-dimensional offset point may be determined in a linear manner according to the coordinates of the reference point corresponding to the object query vector.

[0073] Secondly, the projections of the at least one three-dimensional offset point and the reference point in the multi-view image space are determined, and a second offset is performed on each projection in the multi-view image space to obtain a plurality of two-dimensional offset points.

[0074] The camera parameters can be used to project the three-dimensional offset points and the reference points of the object query vector into the multi-view image space to obtain the projection points in the multi-view image. For each projection point, a second offset is performed to obtain multiple two-dimensional offset points. For each two-dimensional offset point, pixels related to the two-dimensional offset point are searched in each feature map. The features corresponding to the pixels related to the two-dimensional offset point in each feature map are fused.

[0075] Since the object query vector contains rich 3D information (e.g., position, size, and orientation), the 3D offset can be directly generated by applying a linear method to the object query vector itself. Where N q is the number of object query vectors, N 3d is the number of 3D learnable offset points, which can be represented by the following formula:

[0076] Δ 3d =Linear(Q) (3);

[0077] in, is the query vector (object query vector); Linear() is a linear function.

[0078] Due to the lack of two-dimensional information in three-dimensional space, it is difficult to generate a reliable two-dimensional offset based only on the object query vector features. In order to solve this problem, a bidirectional offset is used. The reference point of the object query vector is first offset in three-dimensional space to obtain at least one three-dimensional offset point. The at least one three-dimensional offset point and the reference point are projected into the multi-view image space (two-dimensional space) to obtain multiple two-dimensional projection points in the multi-view image space. For each projection point, an offset is performed in the multi-view image space to obtain multiple two-dimensional offset points. Then, the image features corresponding to the projection of the reference point in the multi-view image space are fused with the object query vector, and the image features of the multiple two-dimensional offset points are fused with the object query vector.

[0079] Given an object query vector reference point First, the reference point is projected into the image space according to the camera parameters to obtain a two-dimensional projection point Then the features at the corresponding positions in all feature layers are sampled and fused together. For simplicity, the process is expressed as the following formula:

[0080]

[0081] Where sampling is a grid sampling operation. Represents multi-scale two-dimensional image features. After obtaining the sampling features After that, the two-dimensional offset It can be generated by a linear method as shown in the following formula:

[0082] Δ 2d =Linear(Q+X img ) (5);

[0083] Based on the bidirectional offset (including 3D offset and 2D offset), the final sampling point can be expressed as follows:

[0084] P 2d =K(P 3d +Δ 3d )+Δ 2d (6);

[0085] in K represents the 3D to 2D projection matrix based on the camera's intrinsic and extrinsic parameters.

[0086] S305, determining the associated image features of each two-dimensional offset point.

[0087] Specifically, the above step S305 includes the following steps:

[0088] First, for each two-dimensional offset point, feature information corresponding to the two-dimensional offset point is determined in the multi-scale feature map.

[0089] Secondly, the feature information corresponding to each of the multi-scale feature maps is fused to obtain the associated image features of the two-dimensional offset point.

[0090] That is, for each sampling point (which can be regarded as a two-dimensional offset point), the corresponding features in each scale feature map are fused to obtain the associated image features of the sampling point. Therefore, the associated image features may include the fusion features of the respective 3D position perception features in the multi-scale feature map. The image features of the object query vector can be increased through the above-mentioned associated image features, and the object query vector can also learn the position information. Fusion of multi-scale features can reduce the amount of calculation.

[0091] S306 , using the associated image features of the plurality of two-dimensional offset points and the associated image features of the projection of the reference point to interact with the object query vector, to update the object query vector.

[0092] The cross-attention algorithm can be used to interact the associated image features of multiple two-dimensional offset points and the associated image features of the projection of the reference point with the object query vector to update the object query vector.

[0093] S307: predicting the object category and location information according to the updated object query vector.

[0094] The above-mentioned object position information includes position information indicating a position in a three-dimensional space, including three-dimensional coordinates and orientation, etc.

[0095] After the object query vector is updated, the object category and location information may be predicted based on the updated object query vector.

[0096] Compared to Figure 2 The embodiment shown in the figure describes that by using bidirectional offsets during decoding, the interactive attention learned through bidirectional offsets can simultaneously learn the offsets of reference points in 2D and 3D space, thereby alleviating the negative impact caused by the inconsistency between the object occupied area in three-dimensional space and two-dimensional image space; promoting accurate and effective interaction between object query vectors and image features; realizing local attention aggregation through learning the offsets of 3D reference points in 2D images, and adapting to feature changes of objects of different distances and sizes by learning the offsets of reference points in 2D and 3D space at the same time. It is helpful to improve the precision and accuracy of target detection results determined according to object query vectors.

[0097] Please refer to Figure 4 , Figure 4 is a schematic diagram of the third embodiment of the present disclosure, such as Figure 4 As shown, the target detection method provided in this embodiment includes the following steps:

[0098] S401: Input the multi-view image into the backbone network and the feature pyramid network, and the feature pyramid network outputs a multi-scale feature map.

[0099] The backbone network may be any neural network for extracting image features, such as a convolutional neural network, etc. The feature pyramid network may enhance the feature map output from the backbone network.

[0100] S402: Determine three-dimensional coordinate information of each scale feature map in space, wherein the three-dimensional coordinate information of each scale feature map includes three-dimensional coordinates of a plurality of points corresponding to each pixel point in the feature map in space.

[0101] The specific implementation of the above step S402 can refer to Figure 2 Step S202 in the illustrated embodiment.

[0102] As an implementation method, the multi-scale feature map and camera parameters can be input into a 3D position generation network, and the 3D position generation network generates three-dimensional coordinates of multiple points corresponding to each pixel in each scale feature map.

[0103] S403: Input each scale feature map and the three-dimensional coordinate information corresponding to the scale feature map to the encoder, and the encoder encodes each scale feature map and the three-dimensional coordinate information corresponding to the scale feature map, and determines the three-dimensional position perception features of the multi-scale feature map according to the encoding results.

[0104] For each scale feature map, an encoder is used to encode the features of each pixel in the feature map and the three-dimensional coordinate information of the pixel to obtain the three-dimensional position perception feature of the pixel.

[0105] As an implementation method, for each scale feature map, an encoder is used to encode the scale feature map and the three-dimensional coordinate information of the scale feature map to obtain the three-dimensional position perception feature corresponding to the scale feature map.

[0106] As an implementation method, the position-aware features corresponding to each scale feature map can be fused to obtain the three-dimensional position-aware features of the fused multi-scale feature map.

[0107] S404: The three-dimensional position-aware features of the multi-scale feature map and the object query vector are input into the decoder, and the deformable cross-attention layer in the decoder determines the associated image features of multiple two-dimensional offset points for each object query vector based on the bidirectional offset; the above-mentioned deformable cross-attention layer performs cross-attention calculation on the associated image features of the multiple two-dimensional offset points and the associated image features of the projection of the reference point and the object query vector, and updates the object query vector according to the cross-attention calculation result.

[0108] Specifically, the process of determining multiple two-dimensional offset points of the query object based on the bidirectional offset in the deformable cross attention layer and determining the associated image features of the two-dimensional offset points can be referred to Figure 3 The relevant parts of the illustrated embodiment are not described in detail here.

[0109] The multiple object query vectors may include query position vectors obtained by encoding multiple positions in space respectively. The Q matrix obtained according to the object query vectors can be considered as raising the question "what is at a certain position?".

[0110] The 3D position-aware features output by the encoder are used to generate key vectors and value vectors. Multiple key vectors and value vectors form the K matrix and the V matrix respectively. The K matrix provides some key information for solving the problems in the Q matrix, and the K matrix can be considered to contain some key features of an object at a certain position.

[0111] Please refer to Figure 5 , Figure 5 A schematic diagram of a network structure for realizing target detection disclosed in the present invention. Figure 5 As shown, the network structure includes a backbone network, a feature pyramid network, a 3D generation network, an encoder and a decoder. The 3D generation network can generate a 3D position map corresponding to a multi-scale feature map. The encoder encodes the multi-scale features and the 3D position map corresponding to the multi-scale feature map to obtain a multi-scale 3D position-aware feature. The above decoder may include a deformable self-attention layer, a deformable cross-attention layer, a feedforward network and a hybrid query merging network. After multiple object query vectors are input into the decoder, they are sent to the deformable self-attention layer, and the deformable self-attention layer performs a self-attention mechanism on the multiple object query vectors. Through the self-attention mechanism processing of the above object query vectors, a global contextual relationship can be established between the object query vectors. During the self-attention mechanism processing, each object query vector interacts with all other object query vectors to obtain a more accurate object query vector representation in the global. The 3D position-aware feature input into the decoder generates a corresponding key vector and value vector in the deformable cross-attention layer. The key vector and the value vector are subjected to a cross-attention mechanism processing with the object query vector.

[0112] In the deformable cross-attention layer, multiple relevant 3D position-aware features can be obtained through a bidirectional offset process to generate their respective key vectors and value vectors, which are processed with the above-mentioned object query object through a cross-attention mechanism to update the query object.

[0113] The most relevant points of the object query vector in the image can be captured through the bidirectional offset process, and the image features corresponding to the most relevant points will be sampled to update the query features through the attention mechanism. Let q index have the mixed query feature Q q ∈R 1×C and the reference point in image space The query element, the deformable interactive attention adopted can be expressed by the following formula:

[0114]

[0115] Where m indexes the attention head; k indexes the sampling point; K is an integer greater than or equal to 2; l indexes the feature level. mlqkRepresents the attention weight of the kth sampling point and the mth attention head in the lth feature layer. Attention weight A mlqk The range of W is [0,1]. m and W m ′ is a learnable parameter. By implicitly modeling the 2D to 3D relationship, Represents multi-scale two-dimensional image features.

[0116] The updated query object vector can fuse the 3D position perception features of the projection point in the image space corresponding to the reference point and the associated image features of the relevant two-dimensional offset points, so that the updated query object includes rich image information.

[0117] Multiple views (such as Figure 1 The image data of the 360° panoramic image 11, image 12, image 13, image 14, image 15 and image 16 are input to Figure 5 The backbone network and feature pyramid network of the proposed method are as follows. The feature pyramid network outputs a multi-scale feature map. The 3D generation network then performs 3D mapping on the multi-scale features to obtain the 3D position information corresponding to the multi-scale feature maps. The encoder encodes the multi-scale feature maps and the corresponding 3D position information to obtain 3D position-aware features. The 3D position-aware features are input into the decoder. The deformable cross-attention layer in the decoder generates a key vector and a value vector based on the above 3D position features. The key vector and the value vector are used to perform a cross-attention mechanism on the object query vector to update the object query vector. The updated object query vector has rich image information, thereby obtaining high accuracy and precision of the object detection results in multiple views.

[0118] Figure 5The provided network includes two important modules, namely, a multi-scale 3D position mapping module and a deformable interactive attention layer module. The deformable interactive attention layer can realize the bidirectional offset learning of the reference point of the object query vector. These two modules are designed to realize implicit 2D to 3D and explicit 3D to 2D position relationship learning at the same time. The network embeds both 2D to 3D implicit relationship and 3D to 2D explicit relationship mining, realizing complementary modeling of the two methods. Different from the previous implicit 2D image to 3D space position relationship modeling, the multi-scale 3D position mapping can model the multi-scale features extracted by the backbone network and FPN in the cone space, generate 3D position relationship, and then accurately encode multi-scale value vectors and key vectors during interactive attention. In the related art, the explicit 3D space to 2D image position relationship modeling focuses on the reference point offset learning in a single space, while the interactive attention based on bidirectional offset learning in the present disclosure can learn the offset of the reference point in 2D and 3D space at the same time, which greatly reduces the negative impact of the inconsistency between the object occupied area in the three-dimensional space and the two-dimensional image space. Therefore, it has a higher detection efficiency when performing target detection.

[0119] S405: In the decoder, determine the object category and position information of the object in the multi-view image according to the updated object query.

[0120] The updated object query vector is input into the feedforward network and the hybrid query merging network, and the above networks are used to determine the object and location information corresponding to the object query vector.

[0121] The above-mentioned position information includes 3D position information and direction information, etc.

[0122] In this embodiment, a backbone network and a feature pyramid network are provided to calculate multi-scale feature maps of multiple views, an encoder is used to encode the scale feature maps and the 3D position information corresponding to the multiple sub-views, a decoder including a deformable cross-attention layer decodes the encoded 3D position-aware features and object query vectors, and outputs object information and 3D position information of objects corresponding to the multi-view images, thereby providing 3D target detection results with higher accuracy at higher efficiency.

[0123] Figure 6 is a schematic diagram according to a fourth embodiment of the present disclosure, Figure 6 As shown, the target detection device 600 provided in this embodiment includes:

[0124] An acquisition unit 601 is used to acquire a multi-scale feature map of a multi-view image;

[0125] A determining unit 602 is used to determine the three-dimensional coordinate information of each scale feature map in space, wherein the three-dimensional coordinate information of each scale feature map includes the three-dimensional coordinates of multiple points corresponding to each pixel point in the scale feature map in space;

[0126] The encoding unit 603 is used to encode each scale feature map and the three-dimensional coordinate information of the scale feature map, and determine the three-dimensional position perception feature of the multi-scale feature map according to the encoding result;

[0127] The decoding unit 604 is used to perform decoding based on the three-dimensional position-aware features of the multi-scale feature map to obtain object categories and position information of objects in the multi-view images.

[0128] In some embodiments, the decoding unit includes a first decoding module 6041, and the first decoding module 6041 is used to:

[0129] Decoding is performed based on the three-dimensional position-aware features of the multi-scale feature map and a plurality of preset object query vectors, where the object query vector includes an encoding vector of a position in three-dimensional space.

[0130] In some embodiments, the first decoding module 6041 includes an offset submodule 6042, a determination submodule 6043, an interaction submodule 6044 and a prediction submodule 6045, wherein:

[0131] An offset submodule 6042 is configured to perform a bidirectional offset on each object query vector according to a reference point of the object query vector in the three-dimensional space to obtain a plurality of two-dimensional offset points of the reference point in the image space; wherein the bidirectional offset includes performing a first offset on the reference point in the three-dimensional space, and performing a second offset on the three-dimensional offset point obtained by the first offset and the projection point of the reference point in the two-dimensional space respectively;

[0132] A determination submodule 6043 is used to determine the associated image features of each two-dimensional offset point;

[0133] An interaction submodule 6044 is used to interact with the object query vector by using the associated image features of the plurality of two-dimensional offset points and the associated image features of the projection of the reference point to update the object query vector;

[0134] The prediction submodule 6045 is used to predict the object category and location information according to the updated object query vector.

[0135] In some embodiments, the offset submodule 6042 is further configured to:

[0136] Performing a first offset in the three-dimensional space on a reference point of the object query vector in the three-dimensional space to determine at least one three-dimensional offset point;

[0137] Determine the projections of the at least one three-dimensional offset point and the reference point in the multi-view image space, and perform a second offset on each projection in the multi-view image space to obtain a plurality of two-dimensional offset points; and the determination submodule 6043 is further used to:

[0138] For each two-dimensional offset point, determining feature information corresponding to the two-dimensional offset point in the multi-scale feature map;

[0139] The feature information corresponding to each of the multi-scale feature maps is fused to obtain the associated image features of the two-dimensional offset point.

[0140] In some embodiments, the decoding unit 604 includes a second decoding module 6046, and the second decoding module 6046 is used to:

[0141] The three-dimensional position-aware features of the multi-scale feature map and the object query vector are input into the decoder, and the deformable cross-attention layer in the decoder determines the associated image features of multiple two-dimensional offset points for each object query vector based on the bidirectional offset; the deformable cross-attention layer performs cross-attention calculation on the associated image features of the multiple two-dimensional offset points and the associated image features of the projection of the reference point of the object query vector and the object query vector, and updates the object query vector according to the cross-attention calculation result;

[0142] In the decoder, object categories and position information of objects in the multi-view images are determined according to the updated object query vector.

[0143] In some embodiments, the acquisition unit 601 includes a first acquisition module 6011, and the first acquisition module 6011 is used to:

[0144] The multi-view image is input into the backbone network and the feature pyramid network, and the feature pyramid network outputs a multi-scale feature map.

[0145] In some embodiments, the determining unit 602 includes a first determining module 6021, and the first determining module 6021 is used to:

[0146] Determine the three-dimensional coordinate information of each scale feature map in the viewing cone space corresponding to the multiple views.

[0147] In some embodiments, the encoding unit 603 includes a first encoding module 6031, and the first encoding module 6031 is used to:

[0148] Each scale feature map and the three-dimensional coordinate information corresponding to the scale feature map are input into an encoder, and the encoder encodes each scale feature map and the three-dimensional coordinate information corresponding to the scale feature map, and determines the three-dimensional position perception features of the multi-scale feature map according to the encoding results.

[0149] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0150] According to an embodiment of the present disclosure, the present disclosure further provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute a solution provided by any of the above embodiments.

[0151] According to an embodiment of the present disclosure, the present disclosure also provides a computer program product, which includes: a computer program, the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device executes the solution provided by any of the above embodiments.

[0152] Figure 7 A schematic block diagram of an example electronic device that can be used to implement an embodiment of the present disclosure is shown. Among them, the electronic device 700 can be a terminal device or a server. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0153] like Figure 7 As shown, the device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0154] A number of components in the device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0155] The computing unit 701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above, such as the generation operation process of the electronic case report. For example, in some embodiments, the generation operation process of the electronic case report may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the generation operation process of the electronic case report described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform the electronic case report generation operation process in any other appropriate manner (eg, by means of firmware).

[0156] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0157] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0158] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0159] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0160] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0161] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0162] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0163] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A target detection method, comprising: Obtain multi-scale feature maps of multi-view images; Determine three-dimensional coordinate information of each scale feature map in space, wherein the three-dimensional coordinate information of each scale feature map includes three-dimensional coordinates of a plurality of points in space corresponding to each pixel point in the scale feature map; Encoding each scale feature map and the three-dimensional coordinate information of the scale feature map, and determining the three-dimensional position perception features of the multi-scale feature map according to the encoding result; Decoding the three-dimensional position-aware features of the multi-scale feature map to obtain object categories and position information of objects in the multi-view image; The decoding is performed based on the three-dimensional position-aware features of the multi-scale feature map, including: For each object query vector, bidirectional offset is performed according to a reference point of the object query vector in the three-dimensional space to obtain a plurality of two-dimensional offset points of the reference point in the image space; wherein the object query vector includes an encoding vector of a position in the three-dimensional space; the bidirectional offset includes performing a first offset on the reference point in the three-dimensional space, and performing a second offset on the three-dimensional offset point obtained by the first offset and the projection point of the reference point in the two-dimensional space respectively; determining an associated image feature for each two-dimensional offset point; Using the associated image features of the plurality of two-dimensional offset points and the associated image features of the projection of the reference point to interact with the object query vector, the object query vector is updated; Object category and location information are predicted based on the updated object query vector.

2. The method according to claim 1, characterized in that: For each object query vector, bidirectional offset is performed according to a reference point of the object query vector in the three-dimensional space to obtain a plurality of two-dimensional offset points of the reference point in the image space, including: Performing a first offset on a reference point of the object query vector in the three-dimensional space to determine at least one three-dimensional offset point; Determining projections of the at least one three-dimensional offset point and the reference point in the multi-view image space, and performing a second offset on each projection in the multi-view image space to obtain a plurality of two-dimensional offset points; and The step of determining the associated image feature of each two-dimensional offset point includes: For each two-dimensional offset point, determining feature information corresponding to the two-dimensional offset point in the multi-scale feature map; The feature information corresponding to each of the multi-scale feature maps is fused to obtain the associated image feature of the two-dimensional offset point.

3. The method according to claim 1, characterized in that The decoding of the three-dimensional position-aware features based on the multi-scale feature map includes: The three-dimensional position-aware features of the multi-scale feature map and the object query vector are input into a decoder, and a deformable cross-attention layer in the decoder determines, for each object query vector, associated image features of a plurality of two-dimensional offset points based on a bidirectional offset; the deformable cross-attention layer performs a cross-attention calculation on the associated image features of the plurality of two-dimensional offset points and the associated image features of the projection of the reference point of the object query vector and the object query vector, and updates the object query vector according to the cross-attention calculation result; In the decoder, object categories and position information of objects in the multi-view images are determined according to the updated object query vector.

4. The method according to claim 1, characterized in that The step of acquiring a multi-scale feature map of a multi-view image comprises: The multi-view image is input into a backbone network and a feature pyramid network, and the feature pyramid network outputs a multi-scale feature map.

5. The method according to claim 1, characterized in that The determining of the three-dimensional coordinate information of each scale feature map in space includes: Determine the three-dimensional coordinate information of each scale feature map in the viewing cone space corresponding to the multiple views.

6. The method according to claim 1, characterized in that The step of encoding each scale feature map and the three-dimensional coordinate information of the scale feature map, and determining the three-dimensional position perception features of the multi-scale feature map according to the encoding result, includes: Each scale feature map and the three-dimensional coordinate information corresponding to the scale feature map are input into an encoder, and the encoder encodes each scale feature map and the three-dimensional coordinate information corresponding to the scale feature map, and determines the three-dimensional position perception features of the multi-scale feature map according to the encoding results.

7. A target detection device, comprising: An acquisition unit, for acquiring a multi-scale feature map of a multi-view image; A determination unit, configured to determine three-dimensional coordinate information of each scale feature map in space, wherein the three-dimensional coordinate information of each scale feature map includes three-dimensional coordinates of a plurality of points in space corresponding to each pixel point in the scale feature map; An encoding unit, used to encode each scale feature map and the three-dimensional coordinate information of the scale feature map, and determine the three-dimensional position perception features of the multi-scale feature map according to the encoding result; A decoding unit, configured to decode based on the three-dimensional position-aware features of the multi-scale feature map to obtain object categories and position information of objects in the multi-view image; The decoding unit includes a first decoding module, which includes an offset submodule, a determination submodule, an interaction submodule and a prediction submodule, wherein: The offset submodule is used to perform bidirectional offset on each object query vector according to a reference point of the object query vector in the three-dimensional space to obtain a plurality of two-dimensional offset points of the reference point in the image space; wherein the bidirectional offset includes performing a first offset on the reference point in the three-dimensional space, and performing a second offset on the three-dimensional offset point obtained by the first offset and the projection point of the reference point in the two-dimensional space respectively; A determination submodule, used to determine the associated image features of each two-dimensional offset point; An interaction submodule, configured to interact with the object query vector by utilizing the associated image features of the plurality of two-dimensional offset points and the associated image features of the projection of the reference point, so as to update the object query vector; The prediction submodule is used to predict the object category and location information based on the updated object query vector.

8. The device according to claim 7, characterized in that The offset submodule is also used for: Performing a first offset on a reference point of the object query vector in the three-dimensional space to determine at least one three-dimensional offset point; Determining projections of the at least one three-dimensional offset point and the reference point in the multi-view image space, and performing a second offset on each projection in the multi-view image space to obtain a plurality of two-dimensional offset points; and The determining submodule is also used for: For each two-dimensional offset point, determining feature information corresponding to the two-dimensional offset point in the multi-scale feature map; The feature information corresponding to each of the multi-scale feature maps is fused to obtain the associated image feature of the two-dimensional offset point.

9. The device according to claim 7, characterized in that The decoding unit includes a second decoding module, and the second decoding module is used for: Inputting the three-dimensional position-aware features of the multi-scale feature map and the object query vector into a decoder, and determining, by a deformable cross-attention layer in the decoder, associated image features of a plurality of two-dimensional offset points for each object query vector based on a bidirectional offset; The deformable cross-attention layer performs cross-attention calculation on the associated image features of the multiple two-dimensional offset points and the associated image features of the projection of the reference point of the object query vector and the object query vector, and updates the object query vector according to the cross-attention calculation result; In the decoder, object categories and position information of objects in the multi-view images are determined according to the updated object query vector.

10. The device according to claim 7, characterized in that The acquisition unit includes a first acquisition module, and the first acquisition module is used for: The multi-view image is input into a backbone network and a feature pyramid network, and the feature pyramid network outputs a multi-scale feature map.

11. The device according to claim 7, characterized in that The determining unit includes a first determining module, and the first determining module is used to: Determine the three-dimensional coordinate information of each scale feature map in the viewing cone space corresponding to the multiple views.

12. The device according to claim 7, characterized in that The encoding unit includes a first encoding module, and the first encoding module is used for: Each scale feature map and the three-dimensional coordinate information corresponding to the scale feature map are input into an encoder, and the encoder encodes each scale feature map and the three-dimensional coordinate information corresponding to the scale feature map, and determines the three-dimensional position perception features of the multi-scale feature map according to the encoding results.

13. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-6.

15. A computer program product, comprising a computer program, which implements the steps of the method according to any one of claims 1 to 6 when executed by a processor.

Images