An image search method for a smart mobile terminal

By employing a multimodal image labeling network architecture and specific convolution operators for image feature extraction on mobile devices, the issues of privacy protection, personalization, and computational efficiency in image search on mobile devices are resolved, achieving an efficient and personalized image search experience.

CN117218336BActive Publication Date: 2025-12-12JIAXING RES INST ZHEJIANG UNIV
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202311284198.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-09-28
Publication Date
2025-12-12
Estimated Expiration
2043-09-28

AI Technical Summary

Technical Problem

Existing image search technologies on mobile devices suffer from insufficient privacy protection, inadequate personalized search, low computational efficiency, and inaccurate feature extraction. They are particularly ineffective when processing specific types of images, and multi-image search tasks are time-consuming, impacting user experience.

Method used

A multimodal image labeling network architecture based on user habits is adopted. It combines single-region convolution operators and Region Act Fun operators to extract ROI region features, uses R-Conv convolution operators and fully connected layers for personalized search, generates image labels based on location, date and weather information, and filters them according to user-input search keywords.

Benefits of technology

It enables efficient and personalized image search on mobile devices, protects user privacy, improves feature extraction accuracy and search speed, and enhances user experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117218336B_ABST
    Figure CN117218336B_ABST
Patent Text Reader

Abstract

The application discloses an image search method for a smart mobile terminal, comprising: acquiring a plurality of images for searching and position, date, time and weather information when the images are collected, and performing gray scale conversion on each image; calculating a score weight matrix corresponding to each image based on the position, date, time and weather information corresponding to the converted image; performing ROI region feature extraction on each image based on a single region convolution operator and a Region Act Fun operator; for each image, performing feature processing on the corresponding image based on the ROI region, and obtaining a label value of the image in combination with the score weight matrix of the image; acquiring a search category keyword input by a user, screening according to the similarity between the search category keyword and the label value of each image, obtaining the confidence of the screened image to the search category keyword through an R-Conv convolution operator and a full connection layer, and performing secondary screening, so as to obtain a search result.
Need to check novelty before this filing date? Find Prior Art

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of big data, image and mobile device, and particularly relates to an image search method for intelligent mobile terminal. BACKGROUND

[0002] In the existing image search technology, cloud servers are usually used to store and process large amounts of image data. This approach not only exposes users' personal information, but also has certain dependence on network environment and server performance. In terms of search speed and accuracy, traditional image search engines mainly rely on pre-defined tags or user-provided keywords for searching, which does not fully consider the influence of user personal habits and multi-modal information (such as location, time, weather, etc.), so the search results often cannot meet the personalized needs of users. In addition, the image feature extraction in the prior art is mainly based on complex convolutional neural network models. Although these models have high accuracy in image recognition, they also bring huge computational load. Especially on mobile devices, due to the relatively limited hardware resources, such models often make it difficult to achieve real-time and efficient image search. In terms of feature extraction, existing methods usually use fixed convolution kernels and activation functions, which are not always ideal for different types and sources of images. These general methods may not accurately capture the most critical feature information when processing specific types of images (such as selfies, food photos, etc.). Furthermore, for multi-image search tasks, existing technologies do not have a good solution to achieve efficient batch processing. Even some methods provide batch search functions, but due to the lack of effective optimization mechanisms, such search processes are usually time-consuming, affecting user experience. Overall, existing image search technologies have certain shortcomings in privacy protection, personalized search, computational efficiency and feature extraction accuracy. These problems limit the widespread application of image search technology on mobile devices and affect user experience and data security. Therefore, it is necessary to develop a new image search system to solve the above problems. SUMMARY

[0003] To solve the above problems, the present application proposes an image search method for intelligent mobile terminal, which can realize fast retrieval of images in mobile devices. The technical solution is as follows:

[0004] Obtain a plurality of images for searching and collect the position, date, time and weather information when the images are collected, and perform gray scale conversion on each image;

[0005] Based on the position, date, time and weather information corresponding to each converted image, calculate the score weight matrix corresponding to the image;

[0006] Based on single-region convolution operator, Region Act Fun operator, ROI region feature extraction is performed on each image;

[0007] For each image, based on the ROI region, feature processing is performed on the corresponding image, and the label value of the image is obtained by combining the score weight matrix of the image;

[0008] The user input search category keyword is obtained, the similarity between the search category keyword and the label value of each image is filtered, the confidence of the search category keyword for the filtered image is obtained through R-Conv convolution operator and full connection layer, and secondary filtering is performed according to the confidence, thereby obtaining the search result.

[0009] Further, the position information is obtained by using the GPS function of the smart phone, the date and time information is obtained from the system of the smart phone, and the weather information is obtained by using the meteorological application in the smart phone.

[0010] Further, based on the position, date, time and weather information, the possible label of the image is given, including:

[0011] The position, date, time and weather information are preprocessed, wherein the position information is normalized by longitude and latitude, the date information is given a week time label and a month time label, the time information is given an intra-day time label, and the weather information is given a weather profile label, a temperature label and a humidity label according to the weather category and temperature, humidity.

[0012] The information obtained by preprocessing is classified by SVM algorithm to obtain the possible label of the image.

[0013] Further, the longitude and latitude normalization processing is:

[0014]

[0015] Wherein, Nla, Nlo are the normalized longitude and latitude coordinates, respectively, and lo, la are the original longitude and latitude coordinates, respectively.

[0016] Further, based on single-region convolution operator, Region Act Fun operator and the possible label of the image, ROI region feature extraction is performed on the image, including:

[0017] Based on single-region convolution operator, the average jump of pixel value under each operator region in the image is calculated;

[0018] Based on the average jump of the pixel value under each operator region, the Region Act Fun operator is used to calculate the minimum pixel value jump set of the first several different pixel averages in the image.

[0019] Based on the minimum pixel value jump set, all operator regions are filtered, and the single-region convolution operator and the Region Act Fun operator are used again to extract the ROI region.

[0020] Further, the single-region convolution operator is represented as follows:

[0021]

[0022] where RSM nom represents the average jump of the pixel value under the current convolution operator region, pix i is the pixel value of the current pixel point, represents the average value of all pixel points under the current operator.

[0023] Further, the Region Act Fun operator is represented as follows:

[0024]

[0025]

[0026] where RSM set is the pixel value average jump set of the smaller single-region convolution operator region, RSM min1 , RSM min2 , RSM min3 , RSM min4 , RSM minN are the single-region convolution operator regions of the first N minimum pixel value average jumps, is the pixel average value of the single-region convolution operator region, and α represents the interval hyperparameter.

[0027] Further, based on the ROI region, the image is processed for features, and the label value of the image is obtained by combining the score weight matrix, including:

[0028] The image is subjected to convolution processing and segmentation operation, and reserved segmentation is performed according to a predetermined proportion.

[0029] The segmented channels are subjected to flattening operation and connected with the ROI region.

[0030] The vector obtained by splicing is subjected to full connection layer and SoftMax layer output, and the score weight matrix is combined for classification calculation to obtain the prediction probability, thereby obtaining the label value of the image.

[0031] Further, the search category keywords and the label values of each image are filtered according to the similarity between them, the confidence of the filtered images to the search category keywords is obtained through an R-Conv convolution operator and a fully connected layer, and secondary filtering is performed according to the confidence, so as to obtain a search result, including:

[0032] The cosine similarity between the search category keywords and the label values of each image is calculated.

[0033] Images with a cosine similarity greater than or equal to a predetermined judgment factor are filtered.

[0034] The filtered images are processed through three R-Conv convolution operators and one fully connected layer to obtain the confidence of the remaining channels of all images under the label to the category θ2, wherein the R-Conv convolution operator is used to segment the channels of the image.

[0035] Secondary filtering is performed according to the confidence θ2, so as to obtain a search result.

[0036] Further, the R-Conv convolution operator is represented as:

[0037] p = μ * p

[0038]

[0039] wherein p is a pixel point of an image feature value, μ is an image regional factor, and σ is a user input category graphic block.

[0040] The technical scheme provided by the embodiment of the application can include the following beneficial effects:

[0041] 1) The application proposes a multi-modal image label giving network architecture based on user habits, which can generate personalized image labels according to multi-modal information such as the location, time and weather of the user's photographing. This innovation realizes a highly personalized image search experience.

[0042] 2) The application introduces an image search algorithm that is globally executed locally, which can complete data storage and search tasks without relying on external servers. This design effectively protects user data privacy.

[0043] 3) The application develops a single-region convolution operator and a region convolution activation function, which are specially designed to extract region vectors of large areas of approximate pixel values in images, realizing accurate extraction of image features that users are most concerned about.

[0044] 4) The present application proposes a convolution algorithm named R-Conv, which can dynamically adjust the weight distribution of the convolution kernel according to the user's search keywords. This innovative algorithm enhances the performance in specific search scenarios.

[0045] It should be understood that the general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0046] The accompanying drawings incorporated in and forming a part of the specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the application.

[0047] Figure 1 is a flow chart of an image search method for a smart mobile terminal according to an embodiment of the present application;

[0048] Figure 2 is an algorithm network architecture diagram of an image search method for a smart mobile terminal according to an embodiment of the present application;

[0049] Figure 3 is a schematic diagram of a smart mobile terminal according to an embodiment of the present application;

[0050] Figure 4 is a block diagram of an image search device for a smart mobile terminal according to an embodiment of the present application;

[0051] Figure 5 is a schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0052] The exemplary embodiments will be described in detail herein below with reference to the drawings. The following description is merely exemplary and explanatory in nature and is not intended to limit the present application, as described above. The same reference numbers in different drawings denote the same or similar elements or features.

[0053] The terms used in the present application are merely for the purpose of describing particular embodiments and are not intended to limit the present application. As used in the present application and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "and / or," as used herein, refer to and encompass any or all possible combinations of one or more of the associated listed items.

[0054] It should be understood that, although the terms first, second, third, etc. can be employed in this application to describe various information, the information should not be limited to these terms. These terms are only used to differentiate one piece of information from another piece of information of the same type. For example, without departing from the scope of the application, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information. Depending on the context, the word "if" as used herein can be interpreted as "when" or "upon determination" or "in response to determining".

[0055] As shown in Figure 1 and Figure 2 The present application provides an image search method for a smart mobile terminal, which can be applied to the CPU processor of various smart mobile terminals (such as smart phones, smart tablets, etc.), as shown in Figure 3 The smart mobile terminal at least includes a data storage and a CPU processor, and can also include a camera, etc. The method can include the following steps:

[0056] Step S1: Obtain a plurality of images for searching and position, date, time and weather information when collecting the images, and perform gray scale conversion on each image;

[0057] In this step, when the user opens the camera application and prepares to take a picture, the system automatically starts the data collection module, uses the GPS function of the mobile phone to obtain the current position information, and synchronously obtains the date and time information through the smart terminal and the NTP server, and obtains the weather information by using the built-in meteorological application of the mobile phone. In addition, all images need to be converted to gray scale to reduce image processing and reduce the difficulty of subsequent processing.

[0058] Step S2: Obtain a score weight matrix corresponding to each image based on the position, date, time and weather information corresponding to the converted image;

[0059] In step S1, the image data taken by the user and the address, date, time, weather information at the time of taking the photograph are obtained, and in this step, the data is first pre-processed, that is, encoded, so that the entire data can be input into the SVM algorithm. For the date and time, first, the time within a day is segmented, and the day is divided into morning (5:00-11:00), noon (11:00-14:00), afternoon (14:00-18:00), evening (18:00-00:00) and early morning (00:00-5:00), and given digital labels 0, 1, 2, 3 and 4 respectively. The date is encoded into two kinds of information, one is given a time within a week label, from Monday to Sunday, given 0, 1, 2, 3, 4, 5, 6 and 7 respectively, and the other is given a month time label, from January to December, given a digital label according to the month. For the address of the user's photograph, the latitude and longitude normalization operation is performed in this application.

[0060] wherein the latitude and longitude normalization can be represented as:

[0061]

[0062] wherein Nla, Nlo are the normalized longitude coordinate and latitude coordinate respectively, and lo, la are the original longitude coordinate and latitude coordinate respectively.

[0063] The user's geographic location coordinates can be converted into a value within the range of [0, 1] by formula (1). Finally, the weather information is encoded as a classification variable, that is, sunny, cloudy, rainy, and the temperature and humidity are classified as low, medium and high.

[0064] In summary, eight features, including the time within a day, the time within a week, the time of the month, the latitude and longitude, the weather profile, the temperature and the humidity, are completed, and the data is arranged in order according to the position x1, x2, x3, x4, x5, x6, x7, x8, which is input into the SVM algorithm network, and a SoftMax layer is calculated to obtain the score weight matrix. Subsequently, the target of the multi-classification task can be determined according to the score weight matrix, such as portrait, food, animal, plant, dress, transportation, etc., and the cross-entropy loss function can be used as the loss function.

[0065] That is, through this step, the image information taken by the user and the feature information including the time within a day, the time within a week, the time of the month, the latitude and longitude, the weather profile, the temperature and the humidity can be input into the SVM algorithm network to give the possible labels to the image taken by the user.

[0066] Step S3: based on a single region convolution operator, Region Act Fun operator, ROI region feature extraction is performed on each image;

[0067] In step S2, given the possible label information of the image taken by the user, the image processing and feature extraction are needed according to the label information in this step, wherein a single region convolution operator is proposed for region determination, and the size of the convolution operator can be 4*4, 8*8, 16*16, which is used to find the region with close pixel values in the image.

[0068] The single region convolution operator is represented as follows:

[0069]

[0070] The RSM nom represents the average jump of the pixel value in the region of the convolution operator, and pix i is the pixel value of the current pixel point. represents the average value of all pixel points under the current operator, and n pixel is the number of pixel points under the current operator.

[0071] The average jump of the pixel value in the region of the operator is calculated by formula (2), and in order to avoid that the combination extracts the background of the photo, a Region Act Fun operator is proposed to perform gentle filtering and selection on the region.

[0072] The Region Act Fun operator is represented as follows:

[0073]

[0074]

[0075] The RSM set is a set of average jumps of small pixel values in the region of the single region convolution operator, and RSM min1 , RSM min2 , RSM min3 , RSM min4 , …, and RSM minN are N single region convolution operators with the smallest average jumps of pixel values, wherein N≥5, and the entire image is covered as much as possible. is the average value of the pixels in the region of the single region convolution operator, and α represents the interval hyperparameter.

[0076] The minimum pixel value jump set of the first N different pixel average values in an image can be calculated by the above formula (3) and formula (4), and all single region convolution operators are filtered, wherein the filtering value is determined according to RSM setSet, for example, the average value of the pixels in the two single-region convolution operator regions is large, respectively corresponding to the portrait and the plant, and the average value of the remaining pixels is much smaller than the two values, so only the average jump of the two values will exceed the range of RSM set , and the average value of the pixels in the remaining single-region convolution operator regions within the filter value is set to 0. The result is compared with the label given in step S2 to ensure the accuracy of the model.

[0077] After passing through a single-region convolution operator with a size of 16*16 and a Region Act Fun, the remaining single-region convolution operators that do not meet the filtering conditions are directly reset to 0, thereby reducing the subsequent calculation amount. In this application, a single-region convolution operator with a size of 9*9 is further used for convolution calculation, and the same Region Act Fun is used for filtering, so as to finally achieve the effect of separating the background from the user's attention area in the image. The pixel values of the remaining all regions are set to 0, thereby forming an image page, which is saved as a γ1 vector.

[0078] Step S4: For each image, performing feature processing on the corresponding image based on the ROI region, and obtaining the label value of the image by combining the score weight matrix of the image;

[0079] In this step, the image taken by the user is generally convolved, and the network layer adopted is Conv, ReLU, and BatchNormal, and the number of layers is four, wherein the size of the Conv operator is 3*3, the interval is 1, and the padding is 1. After four layers of convolution calculation, the image taken by the user is split (Split) according to a 1:2 ratio, and the 2 / 3 channel split out is converted into a vector by Word2Embedding and stored and reserved. The 1 / 3 split channel is flattened (Flatten) and spliced (contact) with the γ1 vector in step S3 to form a δ vector. Then, the δ vector is subjected to a full connection layer operation, and a SoftMax layer is passed through, and the score weight matrix in step S2 is classified and calculated to output the prediction probability, i.e., the label value of the image taken by the user. The image is given a label information, and the label is converted by Word2Embedding and stored.

[0080] Step S5: Obtain the search category keyword input by the user, filter according to the similarity between the search category keyword and the label value of each image, obtain the confidence of the search category keyword for the filtered image through R-Conv convolution operator and full connection layer, and perform secondary filtering according to the confidence, thereby obtaining the search result;

[0081] In the above step, the storage of the image taken by the user and the label given are completed, and in this step, the user input text analysis is first performed, that is, the user input is converted into a vector by using Word2Embedding, and a cosine calculation is performed on the stored image label to view the cosine approximation degree as the confidence of all images under the category of the user input text, and the image screening is performed according to the image confidence determination formula.

[0082] The image confidence determination formula can be expressed as:

[0083] θ1≥ω ⑸

[0084] Wherein, θ1 is the confidence of the initial stored image for the category of the user input, and ω is a judgment factor, which can be set according to actual situation and demand.

[0085] Through the formula, the confidence of all images under the category of the user input exceeding the judgment factor can be output. In addition, in this application, an R-Conv convolution operator is proposed, and the R-Conv convolution calculation is performed on the remaining 2 / 3 channels of the stored image according to the category of the user input in step S4.

[0086] The R-Conv can be expressed as:

[0087] p=μ*p ⑹

[0088]

[0089] Wherein, p is the image feature value pixel point, μ is the image regional factor, and σ is the user input category graphic block (that is, the position of the image in which the content exists in all other images under the keyword category input by the user).

[0090] Through the formula (6) and the formula (7), the R-Conv convolution operator can complete the extraction of the most likely region of the remaining 2 / 3 channels in step S4, wherein whether the image feature value pixel point belongs to the user input category graphic block, that is, the position of the image in which the content exists in all other images under the keyword category input by the user is viewed. For example, the user searches for a selfie, and the key feature face is generally in the center of the image. After such three-layer R-Conv operators and a full connection layer, the confidence θ2 of the remaining channels of all images under the label to the category is output, and the images are screened according to the confidence θ2, and the face selfie related pictures searched by the user are obtained.

[0091] Corresponding to the foregoing embodiments of the image search method for the intelligent mobile terminal, the application also provides embodiments of the image search device for the intelligent mobile terminal.

[0092] Figure 4 is a block diagram of an image search device for a smart mobile terminal according to an exemplary embodiment. Referring to Figure 4 , the device can include:

[0093] an acquisition module 21 configured to acquire a plurality of images for search and position, date, time, and weather information at the time of acquiring the images;

[0094] a calculation module 22 configured to calculate a score weight matrix corresponding to each image based on the position, date, time, and weather information corresponding to the image;

[0095] a feature extraction module 23 configured to perform ROI region feature extraction on each image based on a single region convolution operator and a Region Act Fun operator;

[0096] a label calculation module 24 configured to perform feature processing on a corresponding image based on the ROI region for each image, and obtain a label value of the image in combination with the score weight matrix of the image;

[0097] a search module 25 configured to acquire a search category keyword input by a user, perform filtering based on similarity between the search category keyword and the label value of each image, obtain a confidence degree of the search category keyword with respect to the filtered image through an R-Conv convolution operator and a full connection layer, perform secondary filtering according to the confidence degree, and thus obtain a search result.

[0098] As to the device in the above-described embodiments, the specific manner in which each module performs operations has been described in detail in the embodiments of the method, and thus will not be described in detail here.

[0099] As to the device embodiments, since they basically correspond to the method embodiments, the related parts can be referred to the part of the method embodiments. The device embodiments described above are merely illustrative, and the units described as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, that is, they can be located in one place, or distributed on multiple network units. According to actual needs, some or all of the modules can be selected to achieve the purposes of the present application. Those skilled in the art can understand and implement it without creative labor.

[0100] Correspondingly, the present application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the image search method for a smart mobile terminal as described above. As Figure 5 As shown in the figure, a hardware structure diagram of any data processing capable device where the apparatus is located is provided for the image search method for a smart mobile terminal of the embodiment of the present application. In addition to the processor, the memory and the network interface as shown in the figure, any data processing capable device where the apparatus is located in the embodiment can also include other hardware according to the actual functions of the data processing capable device, which will not be described here. Figure 5 As shown in the figure, a hardware structure diagram of any data processing capable device where the apparatus is located is provided for the image search method for a smart mobile terminal of the embodiment of the present application. In addition to the processor, the memory and the network interface as shown in the figure, any data processing capable device where the apparatus is located in the embodiment can also include other hardware according to the actual functions of the data processing capable device, which will not be described here.

[0101] Correspondingly, the present application also provides a computer readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the image search method for a smart mobile terminal as described above. The computer readable storage medium can be an internal storage unit of any data processing capable device, such as a hard disk or a memory. The computer readable storage medium can also be an external storage device, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. Further, the computer readable storage medium can include both an internal storage unit of any data processing capable device and an external storage device. The computer readable storage medium is used to store the computer program and other programs and data required by the data processing capable device, and can also be used to temporarily store data that has been output or will be output.

[0102] Other embodiments of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. The present application is intended to cover any variations, uses or adaptive changes of the present application following the general principles thereof and including those expressly stated or implied herein.

[0103] It should be understood that the present application is not limited to the precise construction that has been described above and shown in the accompanying drawings, and that various modifications and changes can be effected therein by those skilled in the art without departing from the scope of the application.

Claims

1. An image search method for a smart mobile terminal, characterized by, The method comprises the following steps: acquiring a plurality of images for searching and position, date, time and weather information when the images are collected, and performing gray scale conversion on each image; calculating a score weight matrix corresponding to each image based on the position, date, time and weather information corresponding to the converted image; extracting ROI region features of each image based on a single-region convolution operator and a Region Act Fun operator; for each image, performing feature processing on the corresponding image based on the ROI region, and obtaining a label value of the image in combination with the score weight matrix of the image; acquiring a search category keyword input by a user, screening according to the similarity between the search category keyword and the label value of each image, obtaining a confidence of the search category keyword with respect to the screened image through an R-Conv convolution operator and a fully connected layer, and performing secondary screening according to the confidence, thereby obtaining a search result; wherein the ROI region feature extraction of the image based on the single-region convolution operator, the Region Act Fun operator and the possible label of the image comprises: calculating the average fluctuation of the pixel value under each operator region in the image based on the single-region convolution operator; calculating the minimum pixel value fluctuation set of the first several different pixel average values in the image by using the Region Act Fun operator based on the average fluctuation of the pixel value under each operator region; filtering all operator regions based on the minimum pixel value fluctuation set, and using the single-region convolution operator and the Region Act Fun operator again to extract the ROI region; wherein the R-Conv convolution operator is represented as: , , wherein, is an image feature value pixel point, is an image regional factor, is a user input category graphic block.

2. The method of claim 1, wherein, The position information is acquired by using the GPS function of the smart phone, the date and time information is obtained from the system of the smart phone, and the weather information is acquired by using the meteorological application in the smart phone.

3. The method of claim 1, wherein, Based on the position, date, time and weather information, the possible label of the image is given, which comprises: preprocessing the position, date, time and weather information, wherein the position information is subjected to longitude and latitude normalization processing, the date information is given a week time label and a month time label, the time information is given an intra-day time label, and the weather information is given a weather profile label, a temperature label and a humidity label according to the weather category and the temperature and humidity conditions; performing multi-classification task on the information obtained by preprocessing by using the SVM algorithm to obtain the possible label of the image.

4. The method of claim 3, wherein, The longitude and latitude normalization processing is as follows: , wherein, respectively normalized longitude and latitude coordinates, respectively original longitude and latitude coordinates.

5. The method of claim 1, wherein, The single-region convolution operator is represented as follows: , wherein, represents the average jump of pixel values under the region of the current convolution operator, is the pixel value of the current pixel point, represents the average value of all pixel points under the current operator.

6. The method of claim 1, wherein, The Region Act Fun operator is represented as follows: , , wherein, is a set of pixel value average jumps of small regions of the single-region convolutional operator, are regions of the single-region convolutional operator of the first N smallest pixel value average jumps, respectively, is the pixel average value of the single-region convolutional operator region, is denoted as an interval hyperparameter.

7. The method of claim 1, wherein, The feature processing of the image based on the ROI region in combination with the score weight matrix to obtain the label value of the image comprises: performing convolution processing and segmentation operation on the image, and performing reserved segmentation according to a predetermined proportion; performing flattening operation on the segmented channel and connecting the flattened channel with the ROI region; performing classification calculation on the output of the vector obtained by splicing through the fully connected layer and the SoftMax layer in combination with the score weight matrix to obtain a prediction probability, thereby obtaining the label value of the image.

8. The method of claim 1, wherein, According to the similarity between the search category keyword and the label value of each image, the confidence of the screened image to the search category keyword is obtained through an R-Conv convolution operator and a full connection layer, secondary screening is performed according to the confidence, and a search result is obtained, including: Cosine similarity between the search category keyword and the label value of each image is calculated; Images with cosine similarity greater than or equal to a predetermined judgment factor are screened. The screened images are processed by three layers of R-Conv convolution operators and one fully connected layer to obtain the confidence of the remaining channel of all images under the label to the class wherein the R-Conv convolution operator is used for segmenting the channel of the image According to the confidence of high and low to carry out secondary screening, thereby obtaining the search results.

Citation Information

Patent Citations

  • Image texture feature extraction method based on mean jitter transform

    CN109002828A

  • Central point network multi-target detection method based on similarity matrix

    CN111523553A