Image processing method and device, storage medium and electronic equipment

By using the significance feature extraction module composed of ResNet network and pooling units in the visual position recognition model, the problem of high computational complexity of traditional multi-layer convolution module is solved, and the efficient extraction of significance information and rapid generation of target vectors is achieved. Effect.

CN119027782APending Publication Date: 2024-11-26NEUSOFT REACH AUTOMOBILE TECH (SHENYANG) CO LTD
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202411124119.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-15
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

In the traditional visual position recognition model, the significance aggregator uses a multi-layer convolution module for feature extraction, resulting in high computational complexity and a long time to extract significance information in the query image.

Method used

The ResNet network is used to combine the significance feature extraction module and the semantic feature extraction module to extract features through multiple pooling units, avoiding multiplication calculation of multilayer convolution modules and improving extraction efficiency.

Benefits of technology

It realizes efficient extraction of significance information in the query image, quickly generates target vectors for the query image, and reduces the computational complexity.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119027782A_ABST
    Figure CN119027782A_ABST
Patent Text Reader

Abstract

The invention discloses an image processing method and device, a storage medium and electronic equipment, and relates to the technical field of artificial intelligence. The method comprises the steps of obtaining a query image; inputting the query image into the ResNet network to obtain a first feature vector and a second feature vector; inputting the first feature vector into a saliency feature extraction module to obtain a third feature vector; inputting the second feature vector into a semantic feature extraction module to obtain a fourth feature vector; and generating a target vector corresponding to the query image based on the third feature vector and the fourth feature vector. The saliency feature extraction module is composed of a plurality of pooling units, the pooling algorithm in the pooling units does not have calculation parameters or multiplication calculation, the saliency features in the query image can be efficiently extracted, and the target vector corresponding to the query image can be efficiently generated based on the saliency features.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an image processing method, device, storage medium and electronic device. Background Art

[0002] Salient information is the most noticeable or recognizable area or feature in an image. Salient information plays an important role in visual location recognition tasks.

[0003] In the field of autonomous driving technology, the saliency aggregator in the visual location recognition model is usually used to determine the saliency information in the query image, and then generate the feature representation of the query image based on the saliency features. However, since the saliency aggregator in the traditional solution usually uses a multi-layer convolution module to extract the saliency between features, the computational complexity is high. When the query image is input into the traditional visual location recognition model, it takes a long time to wait before the saliency information in the query image can be extracted and the feature representation of the query image can be generated. Summary of the invention

[0004] Based on the above problems, the present application provides an image processing method to improve the efficiency of generating saliency information in a query image and to improve the efficiency of generating a target vector of the query image based on the saliency information.

[0005] The embodiments of the present application disclose the following technical solutions:

[0006] In a first aspect, the present application provides an image processing method, which is applied to a saliency coding visual position recognition model, wherein the saliency coding visual position recognition model includes a ResNet network, a saliency feature extraction module, and a semantic feature extraction module, and the method includes:

[0007] Get a query image;

[0008] Inputting the query image into the ResNet network to obtain a first eigenvector and a second eigenvector;

[0009] Inputting the first feature vector into the significant feature extraction module to obtain a third feature vector; the significant feature extraction module is composed of multiple pooling units;

[0010] Inputting the second feature vector into the semantic feature extraction module to obtain a fourth feature vector;

[0011] A target vector corresponding to the query image is generated based on the third eigenvector and the fourth eigenvector.

[0012] In an optional implementation, the step of inputting the first feature vector into the significant feature extraction module to obtain a third feature vector includes:

[0013] Performing Euclidean norm processing on the first eigenvector to obtain a first reference vector;

[0014] Performing average pooling processing on the first reference vector to obtain a second reference vector;

[0015] generating a third reference vector based on the first reference vector, the second reference vector and the first feature vector;

[0016] The third reference vector is subjected to global pooling processing after passing through the fully connected layer to obtain a fourth reference vector;

[0017] The Euclidean norm processing is performed on the fourth reference to obtain the third eigenvector.

[0018] In an optional implementation, the ResNet network has N convolution blocks; N is an integer greater than or equal to 2 and less than or equal to 5; and inputting the query image into the ResNet network to obtain a first feature vector and a second feature vector includes:

[0019] The first feature vector is generated by the first N-1 convolution blocks in the ResNet network; and the second feature vector is generated by the N convolution blocks in the ResNet network.

[0020] In an optional implementation, after obtaining the second feature vector, the method further includes:

[0021] Inputting the second feature vector into the significant feature extraction module to generate a fifth feature vector;

[0022] The step of generating a target vector corresponding to the query image based on the third feature vector and the fourth feature vector includes:

[0023] A target vector corresponding to the query image is generated based on the third eigenvector, the fourth eigenvector and the fifth eigenvector.

[0024] In an optional implementation, the step of inputting the second feature vector into the significant feature extraction module to generate a fifth feature vector includes:

[0025] Performing the Euclidean norm processing on the second eigenvector to obtain a first reference vector;

[0026] Performing the average pooling process on the first reference vector to obtain a second reference vector;

[0027] Generate a third reference vector based on the first reference vector, the second reference vector and the second eigenvector;

[0028] After the third reference vector passes through the fully connected layer, the global pooling process is performed to obtain a fourth reference vector;

[0029] The fourth reference vector is processed by the Euclidean norm to obtain the fifth eigenvector.

[0030] In an optional implementation, after acquiring the target vector, the method further includes:

[0031] Querying from an image database an image corresponding to a feature vector having the highest similarity to the target vector as the target image;

[0032] The spatial position corresponding to the target image is used as the spatial position corresponding to the query image.

[0033] In a second aspect, the present application provides an image processing device, which is applied to a saliency coding visual position recognition model, wherein the saliency coding visual position recognition model includes a ResNet network, a saliency feature extraction module, and a semantic feature extraction module, and the device includes:

[0034] A query image acquisition module, used to acquire a query image;

[0035] An initial feature vector acquisition module, used for inputting the query image into the ResNet network to obtain a first feature vector and a second feature vector;

[0036] A third feature vector acquisition module, used for inputting the first feature vector into the significant feature extraction module to obtain a third feature vector; the significant feature extraction module is composed of multiple pooling units;

[0037] a fourth feature vector acquisition module, configured to input the second feature vector into the semantic feature extraction module to obtain a fourth feature vector;

[0038] A target vector acquisition module is used to generate a target vector corresponding to the query image based on the third feature vector and the fourth feature vector.

[0039] In an optional implementation, the third feature vector acquisition module includes:

[0040] A first reference vector acquisition unit, configured to perform Euclidean norm processing on the first eigenvector to obtain a first reference vector;

[0041] A second reference vector acquisition unit, configured to perform average pooling processing on the first reference vector to obtain a second reference vector;

[0042] A third reference vector acquiring unit, configured to generate a third reference vector based on the first reference vector, the second reference vector and the first feature vector;

[0043] a fourth reference vector acquisition unit, configured to perform global pooling processing on the third reference vector after the third reference vector passes through the fully connected layer to obtain a fourth reference vector;

[0044] The third eigenvector obtaining unit is configured to perform the Euclidean norm processing on the fourth reference to obtain the third eigenvector.

[0045] A third aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method introduced in any implementation manner of the first aspect.

[0046] A fourth aspect of the present application provides an electronic device, including:

[0047] a memory having a computer program stored thereon;

[0048] A processor is used to execute the computer program in the memory to implement the steps of the method introduced in any implementation manner of the first aspect.

[0049] Compared with the prior art, this application has the following beneficial effects:

[0050] The present application discloses an image processing method, which is applied to a saliency-encoded visual position recognition model, wherein the saliency-encoded visual position recognition model includes a ResNet network, a saliency feature extraction module, and a semantic feature extraction module. The method includes: obtaining a query image; inputting the query image into the ResNet network to obtain a first feature vector and a second feature vector; inputting the first feature vector into the saliency feature extraction module to obtain a third feature vector; inputting the second feature vector into the semantic feature extraction module to obtain a fourth feature vector; and generating a target vector corresponding to the query image based on the third feature vector and the fourth feature vector.

[0051] The significant feature extraction module in the present application is composed of multiple pooling units. The pooling algorithm in the pooling unit has no calculation parameters and no multiplication calculations. It can efficiently extract the significant features in the query image and solve the problem of high computational complexity of the significant feature extraction between the multi-layer convolution module in the traditional solution. After the query image is input into the significant recognition position recognition model in the present solution, the significant information in the query image can be efficiently extracted, and then the target vector corresponding to the query image can be quickly obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0053] Figure 1 A flowchart of an image processing method provided in an embodiment of the present application;

[0054] Figure 2 A schematic diagram of a query image processed by a saliency coding visual position recognition model provided in an embodiment of the present application;

[0055] Figure 3 A schematic diagram of a significant feature extraction module provided in an embodiment of the present application processing a first feature vector;

[0056] Figure 4 A schematic diagram of another saliency coding visual position recognition model provided in an embodiment of the present application processing a query image;

[0057] Figure 5 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0058] Salient information is the most noticeable or recognizable area or feature in an image. Salient information plays an important role in visual location recognition tasks.

[0059] In the field of autonomous driving technology, the saliency aggregator in the saliency encoding visual location recognition model is usually used to determine the saliency information in the query image. However, since the saliency aggregator in the traditional solution usually uses a multi-layer convolution module to extract the saliency between features, the computational complexity is high. When the query image is input into the traditional saliency recognition location recognition model, it takes a long time to wait before the saliency information in the query image is generated.

[0060] Therefore, how to improve the speed of generating saliency information in the query image and obtain the feature representation of the query image based on the saliency information has become a technical problem that needs to be solved urgently.

[0061] The present application discloses an image processing method, which is applied to a saliency-encoded visual position recognition model, wherein the saliency-encoded visual position recognition model includes a ResNet network, a saliency feature extraction module, and a semantic feature extraction module. The method includes: obtaining a query image; inputting the query image into the ResNet network to obtain a first feature vector and a second feature vector; inputting the first feature vector into the saliency feature extraction module to obtain a third feature vector; inputting the second feature vector into the semantic feature extraction module to obtain a fourth feature vector; and generating a target vector corresponding to the query image based on the third feature vector and the fourth feature vector.

[0062] The significant feature extraction module in the present application is composed of multiple pooling units. The pooling algorithm in the pooling unit has no calculation parameters and no multiplication calculations. It can efficiently extract the significant features in the query image and solve the problem of high computational complexity of the significant feature extraction between the multi-layer convolution module in the traditional solution. When the query image is input into the significant recognition position recognition model in the present solution, the significant information in the query image can be efficiently extracted, and then the feature representation of the query image, that is, the target vector corresponding to the query image, can be quickly obtained.

[0063] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0064] Figure 1 A flowchart of an image processing method provided in an embodiment of the present application. Figure 1 As shown, the image processing method in this application includes:

[0065] S101, obtaining a query image.

[0066] The query image in this application can be a photo taken by a vehicle-mounted camera.

[0067] The query image in this application may contain a variety of information such as road environment, other vehicles, pedestrians and non-motor vehicles, traffic lights and signs, obstacles and dangerous situations, as well as natural environment and weather conditions.

[0068] S102, inputting the query image into the ResNet network to obtain a first eigenvector and a second eigenvector.

[0069] The image processing method in the present application is applied to a saliency coding visual position recognition model. The saliency coding visual position recognition model in the present application includes a ResNet network, a saliency feature extraction module and a semantic feature extraction module.

[0070] Figure 2 A schematic diagram of a query image processed by a saliency coding visual position recognition model provided in an embodiment of the present application. Figure 2 As shown, the processing process after the query image is input into the saliency encoding visual position recognition model in this application is described in detail.

[0071] The ResNet network is a deep neural network structure of great significance. It solves the difficulties in deep network training by introducing residual connections and achieves excellent performance in multiple computer vision tasks.

[0072] The ResNet network in this application has N convolutional blocks; where N is an integer greater than or equal to 2 and less than or equal to 5.

[0073] Specifically, after the query image is input into the ResNet network in the present application, the first feature vector is generated through the first N-1 convolution blocks in the ResNet network in the present application; and the second feature vector is generated through the N convolution blocks in the ResNet network.

[0074] Figure 2 The ResNet network in has 5 convolutional blocks. The query image passes through the first 4 convolutional blocks of the ResNet network (i.e. Figure 2 After processing by Conv1, Conv2, Conv3 and Conv4 in the ResNet network, the first feature vector is generated; the query image passes through the first 5 convolution blocks of the ResNet network (i.e. Figure 2 After processing by Conv1, Conv2, Conv3, Conv4 and Conv5 in , the second feature vector is generated.

[0075] Among them, the first eigenvector: F conv4 ∈C1×H / 16 ×W / 16 ; It means that the number of channels of the first eigenvector is C1, the first eigenvector compresses the height (H) of the query image to 1 / 16 of the original, and the first eigenvector compresses the width (W) of the query image to 1 / 16 of the original.

[0076] Among them, the second eigenvector: F conv5 ∈C2×H / 32 ×W / 32; This means that the number of channels of the second eigenvector is C2, the second eigenvector compresses the height (H) of the query image to 1 / 32 of the original, and the first eigenvector compresses the width (W) of the query image to 1 / 32 of the original.

[0077] S103: Input the first feature vector into the significant feature extraction module to obtain a third feature vector.

[0078] After the first eigenvector is dimensionally aligned, the number of channels changes from C1 to C2. The first eigenvector after dimension alignment is: F conv4 ∈C2×H / 16 ×W / 16 In this application, it is the first eigenvector after dimensional alignment, i.e., F conv4 ∈C2×H / 16 ×W / 16 Input to the salient feature extraction module.

[0079] The salient feature extraction module in this application is composed of multiple pooling units. Figure 3 A schematic diagram of a significant feature extraction module provided in an embodiment of the present application processing a first feature vector.

[0080] Combination Figure 3 As shown, the first eigenvector after dimension alignment is input to Figure 3 After the significant feature extraction module in , the first feature vector is first processed by Euclidean norm (L2-Norm) to obtain the first reference vector; then the first reference vector is processed by average pooling to obtain the second reference vector; the first reference vector is subtracted from the second reference vector bit by bit, and then the first feature vector is residually connected and bitwise added to obtain the third reference vector; the third reference vector first passes through a fully connected layer with Dropout, and then undergoes global pooling to obtain the fourth reference vector; the fourth reference vector is processed by Euclidean norm to obtain the third feature vector, that is, The number of channels of the third eigenvector is C2.

[0081] Combination Figure 3 As shown, the significant feature extraction module in the present application is composed of an average pooling unit and a global pooling unit. In the present application, the significant feature extraction module is used to extract the significant features in the first feature vector, replacing the traditional solution of using a multi-layer convolution module to extract the significance between features. Since there are no calculation parameters and no multiplication calculations in the pooling algorithm, the computational complexity of the pooling operation is low, which can solve the problem of high computational complexity of the multi-layer convolution module in the traditional solution for extracting the significance between features and the problem of low efficiency of extracting significant information.

[0082] S104: Input the second feature vector into the semantic feature extraction module to obtain a fourth feature vector.

[0083] The second feature vector is input into the semantic feature extraction module to extract the semantic information of the second feature vector and generate the fourth feature vector, i.e. The number of channels of the fourth eigenvector is C2.

[0084] It is understandable that convolutional neural networks, recurrent neural networks and attention mechanisms can all be used as semantic feature extraction modules. The specific structure of the semantic feature extraction module is not limited in this application.

[0085] S105: Generate a target vector corresponding to the query image based on the third eigenvector and the fourth eigenvector.

[0086] In getting the fourth eigenvector After that, the fourth eigenvector is processed by Euclidean norm and then compared with the third eigenvector Add bit by bit to get the target vector corresponding to the query image.

[0087] See also Figure 4 , when the computing power is sufficient, in order to further extract the significant features in the query image, after obtaining the second feature vector, the second feature vector can be input into the significant feature extraction module to generate a fifth feature vector.

[0088] Specifically, the first feature vector is input into the significant feature extraction module to generate the third feature vector; the second feature vector is input into the semantic feature extraction module to generate the fourth feature vector; the second feature vector is input into the significant feature extraction module to generate the fifth feature vector.

[0089] The fourth eigenvector With the fifth eigenvector After bitwise addition, the Euclidean norm is processed, and the vector obtained after the Euclidean norm processing is added to the third eigenvector Add bit by bit to get the target vector corresponding to the query image

[0090] See also Figure 3 The process in which the significant feature extraction module processes the first feature vector to obtain the third feature vector is described, and the process in which the significant feature extraction module processes the second feature vector to obtain the fifth feature vector is described.

[0091] Specifically, the second feature vector is input into the significant feature extraction module to generate the fifth feature vector in the following specific process: for the second feature vector, i.e., F conv5 ∈C2×H / 32 ×W / 32The first reference vector is processed by Euclidean norm to obtain the first reference vector; the second reference vector is processed by average pooling to obtain the second reference vector; the first reference vector is subtracted from the second reference vector bit by bit, and then the residual vector and the second eigenvector are added bit by bit to obtain the third reference vector; the third reference vector passes through a fully connected layer with Dropout, and then undergoes global pooling to obtain the fourth reference vector; the fourth reference vector is processed by Euclidean norm to obtain the fifth eigenvector, that is, The number of channels of the fifth eigenvector is C2.

[0092] It can be understood that the target feature vector corresponding to the query image obtained by the method in the aforementioned embodiment can represent the comprehensive information generated by the saliency information and semantic information of the query image.

[0093] In an optional implementation, after the target feature vector is obtained, an image corresponding to a feature vector having the highest similarity to the target vector may be queried from an image database as the target image.

[0094] The image database is a database that stores a large number of road environment images on which vehicles travel. It is understandable that the road environment images include data on vehicles, pedestrians, non-motor vehicles, traffic lights, road signs, and obstacles on the road, and even natural environment information and weather conditions when the road images were taken.

[0095] For each image in the image database, the visual position recognition model mentioned in the previous embodiment of the present application is also used to generate a feature vector for each image. It can be understood that the feature vector of each image in the image database has the same dimension and similar expression form as the target feature vector of the query image.

[0096] After determining the target vector of the query image, the similarity between the target vector and the feature vector of each image in the image database is calculated using similarity calculation methods such as cosine similarity or Euclidean distance. According to the similarity calculation results, the images in the image database are sorted, and the image corresponding to the feature vector with the largest similarity value to the target feature vector is obtained as the target image.

[0097] It is understandable that for each image in the image database, the spatial position corresponding to the image is known. Therefore, after determining the target image, the spatial position corresponding to the target image can be used as the spatial position corresponding to the query image.

[0098] Since the query image is an image of the vehicle captured by an onboard camera or detector, once the spatial position of the query image is determined, the current position of the vehicle is determined.

[0099] Based on the image processing method disclosed in the aforementioned embodiment, the present application also discloses an image processing device. Figure 5 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application. Figure 5 As shown, the image processing device 500 in the present application includes:

[0100] A query image acquisition module 501 is used to acquire a query image;

[0101] An initial feature vector acquisition module 502 is used to input the query image into the ResNet network to obtain a first feature vector and a second feature vector;

[0102] A third feature vector acquisition module 503, used to input the first feature vector into the significant feature extraction module to obtain a third feature vector; the significant feature extraction module is composed of multiple pooling units;

[0103] A fourth feature vector acquisition module 504, configured to input the second feature vector into the semantic feature extraction module to obtain a fourth feature vector;

[0104] The target vector acquisition module 505 is used to generate a target vector corresponding to the query image based on the third eigenvector and the fourth eigenvector.

[0105] In an optional implementation, the third feature vector acquisition module 503 includes:

[0106] A first reference vector acquisition unit, configured to perform Euclidean norm processing on the first eigenvector to obtain a first reference vector;

[0107] A second reference vector acquisition unit, configured to perform average pooling processing on the first reference vector to obtain a second reference vector;

[0108] A third reference vector acquiring unit, configured to generate a third reference vector based on the first reference vector, the second reference vector and the first feature vector;

[0109] a fourth reference vector acquisition unit, configured to perform global pooling processing on the third reference vector after the third reference vector passes through the fully connected layer to obtain a fourth reference vector;

[0110] The third eigenvector obtaining unit is configured to perform the Euclidean norm processing on the fourth reference to obtain the third eigenvector.

[0111] In an optional implementation, the initial feature vector acquisition module 502 includes:

[0112] An initial feature vector acquisition unit is used to generate the first feature vector through the first N-1 convolution blocks in the ResNet network; and to generate the second feature vector through the N convolution blocks in the ResNet network.

[0113] In an optional implementation, the image processing apparatus 500 further includes:

[0114] a fifth feature vector acquisition module, configured to input the second feature vector into the significant feature extraction module to generate a fifth feature vector;

[0115] A high-order target vector acquisition module is used to generate a target vector corresponding to the query image based on the third eigenvector, the fourth eigenvector and the fifth eigenvector.

[0116] In an optional implementation, the fifth feature vector acquisition module includes:

[0117] A first reference vector acquisition unit, configured to perform the Euclidean norm processing on the second eigenvector to obtain a first reference vector;

[0118] A second reference vector acquisition unit, configured to perform the average pooling process on the first reference vector to obtain a second reference vector;

[0119] A third reference vector acquiring unit, configured to generate a third reference vector based on the first reference vector, the second reference vector and the second eigenvector;

[0120] a fourth reference vector acquisition unit, configured to perform the global pooling process on the third reference vector after the third reference vector passes through the fully connected layer, so as to obtain a fourth reference vector;

[0121] The fifth eigenvector obtaining unit is configured to perform the Euclidean norm processing on the fourth reference vector to obtain the fifth eigenvector.

[0122] In an optional implementation, the image processing apparatus 500 further includes:

[0123] A target image acquisition module is used to query from an image database an image corresponding to a feature vector having the highest similarity to the target vector as the target image;

[0124] The spatial position determination module is used to use the spatial position corresponding to the target image as the spatial position corresponding to the query image.

[0125] Based on the image processing method and device provided in the aforementioned embodiments, the present application also provides a computer-readable storage medium, on which a computer program is stored, which, when executed by a processor, implements some or all of the steps in the aforementioned image processing method.

[0126] Based on the image processing method and apparatus provided in the above embodiments, the present application further provides an electronic device, including:

[0127] a memory having a computer program stored thereon;

[0128] The processor is used to execute the computer program in the memory to implement part or all of the steps in the image processing method provided in the above embodiment.

[0129] It should be noted that each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described above is merely schematic, in which the unit described as a separate component may or may not be physically separated, and the component prompted as a unit may or may not be a physical unit, that is, it may be located in one place, or it may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative work.

[0130] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An image processing method, It is characterized in that The method is applied to a saliency coding visual position recognition model, wherein the saliency coding visual position recognition model includes a ResNet network, a saliency feature extraction module and a semantic feature extraction module, and the method includes: Get a query image; Inputting the query image into the ResNet network to obtain a first eigenvector and a second eigenvector; Inputting the first feature vector into the significant feature extraction module to obtain a third feature vector; the significant feature extraction module is composed of multiple pooling units; Inputting the second feature vector into the semantic feature extraction module to obtain a fourth feature vector; A target vector corresponding to the query image is generated based on the third eigenvector and the fourth eigenvector.

2. The method according to claim 1, It is characterized in that The step of inputting the first feature vector into the significant feature extraction module to obtain a third feature vector comprises: Performing Euclidean norm processing on the first eigenvector to obtain a first reference vector; Performing average pooling processing on the first reference vector to obtain a second reference vector; generating a third reference vector based on the first reference vector, the second reference vector and the first feature vector; The third reference vector is subjected to global pooling processing after passing through the fully connected layer to obtain a fourth reference vector; The Euclidean norm processing is performed on the fourth reference to obtain the third eigenvector.

3. The method according to claim 1, It is characterized in that The ResNet network has N convolution blocks; N is an integer greater than or equal to 2 and less than or equal to 5; the query image is input into the ResNet network to obtain a first feature vector and a second feature vector, including: The first feature vector is generated by the first N-1 convolution blocks in the ResNet network; and the second feature vector is generated by the N convolution blocks in the ResNet network.

4. The method according to any one of claims 1 to 3, It is characterized in that After obtaining the second feature vector, the method further includes: Inputting the second feature vector into the significant feature extraction module to generate a fifth feature vector; The step of generating a target vector corresponding to the query image based on the third feature vector and the fourth feature vector includes: A target vector corresponding to the query image is generated based on the third eigenvector, the fourth eigenvector and the fifth eigenvector.

5. The method according to claim 4, It is characterized in that The step of inputting the second feature vector into the significant feature extraction module to generate a fifth feature vector comprises: Performing the Euclidean norm processing on the second eigenvector to obtain a first reference vector; Performing the average pooling process on the first reference vector to obtain a second reference vector; Generate a third reference vector based on the first reference vector, the second reference vector and the second eigenvector; After the third reference vector passes through the fully connected layer, the global pooling process is performed to obtain a fourth reference vector; The fourth reference vector is processed by the Euclidean norm to obtain the fifth eigenvector.

6. The method according to claim 1, It is characterized in that After obtaining the target vector, the method further includes: An image corresponding to a feature vector having the highest similarity to the target vector is retrieved from an image database as the target image; The spatial position corresponding to the target image is used as the spatial position corresponding to the query image.

7. An image processing device, It is characterized in that The device is applied to a saliency coding visual position recognition model, the saliency coding visual position recognition model includes a ResNet network, a saliency feature extraction module and a semantic feature extraction module, and the device includes: A query image acquisition module, used for acquiring a query image; An initial feature vector acquisition module, used for inputting the query image into the ResNet network to obtain a first feature vector and a second feature vector; A third feature vector acquisition module, used for inputting the first feature vector into the significant feature extraction module to obtain a third feature vector; the significant feature extraction module is composed of multiple pooling units; a fourth feature vector acquisition module, configured to input the second feature vector into the semantic feature extraction module to obtain a fourth feature vector; A target vector acquisition module is used to generate a target vector corresponding to the query image based on the third feature vector and the fourth feature vector.

8. The device according to claim 7, It is characterized in that The third feature vector acquisition module includes: A first reference vector acquisition unit, configured to perform Euclidean norm processing on the first eigenvector to obtain a first reference vector; A second reference vector acquisition unit, configured to perform average pooling processing on the first reference vector to obtain a second reference vector; A third reference vector acquiring unit, configured to generate a third reference vector based on the first reference vector, the second reference vector and the first feature vector; a fourth reference vector acquisition unit, configured to perform global pooling processing on the third reference vector after the third reference vector passes through the fully connected layer to obtain a fourth reference vector; The third eigenvector obtaining unit is configured to perform the Euclidean norm processing on the fourth reference to obtain the third eigenvector.

9. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the program is executed by a processor, the steps of the method described in any one of claims 1 to 6 are implemented.

10. An electronic device, It is characterized in that include: a memory having a computer program stored thereon; A processor, configured to execute the computer program in the memory to implement the steps of the method according to any one of claims 1 to 6.

Citation Information

Patent Citations

  • Image retrieving method based on salient region

    CN107239565A

  • Saliency target detection method for guiding residual attention based on global information

    CN112329800A

  • Image processing method and device based on artificial intelligence, and electronic equipment

    CN113821667A