Feature extraction model training method, information recommendation method, device and apparatus

Abstract

The present disclosure provides a feature extraction model training method and device, an information recommendation method and device, an electronic device and a computer readable storage medium, and relates to the technical field of image processing. The feature extraction model training method comprises: training a first model to be trained based on a first training image set, the first model comprising a first feature extraction module and a second feature extraction module connected in cascade, the first feature extraction module comprising a point convolution module and a deep convolution module connected in cascade; replacing the second feature extraction module in the trained first model with a feature mapping module to obtain a second model; training the second model through the first training image set, fusing the point convolution module and the deep convolution module of the first feature extraction module in the trained second model, and determining the fused second model as a target model. The present disclosure improves the accuracy of information recommendation.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, specifically to a training method for a feature extraction model, an information recommendation method, a training device for a feature extraction model, an information recommendation device, an electronic device, and a computer-readable storage medium. Background Technology

[0002] With the gradual development of artificial intelligence technology, machine learning, as the core of artificial intelligence, is the fundamental way to enable computers to possess intelligence. Through machine learning, neural network models that meet application requirements can be trained. Nowadays, QR code scanning is widely used in various scenarios, such as location codes and payment codes. However, in many scenarios, there are interfering factors when scanning QR codes, causing too many irrelevant features to be extracted, affecting scanning efficiency and accuracy.

[0003] Existing feature extraction models suffer from inaccurate feature extraction when extracting image features, which affects scanning efficiency and accuracy to some extent, and consequently impacts the user experience. Summary of the Invention

[0004] The purpose of this disclosure is to provide a training method for a feature extraction model, an information recommendation method, a training device for a feature extraction model, an information recommendation device, an electronic device, and a computer-readable medium, thereby improving the accuracy of the feature extraction model to at least a certain extent.

[0005] According to a first aspect of this disclosure, a method for training a feature extraction model is provided, comprising: training a first model to be trained based on a first training image set, the first model including a cascaded first feature extraction module and a second feature extraction module, the first feature extraction module including a cascaded point convolution module and a depth convolution module; replacing the second feature extraction module in the trained first model with a feature mapping module to obtain a second model; training the second model using the first training image set, fusing the point convolution module and the depth convolution module of the first feature extraction module in the trained second model, and determining the fused second model as the target model.

[0006] According to a second aspect of this disclosure, an information recommendation method is provided, comprising: acquiring a collected image, and performing object detection on the collected image based on an object detection model to obtain a graphic code detection result; if it is determined from the graphic code detection result that the collected image includes a graphic code, performing feature extraction on the collected image based on a target model to obtain a feature extraction result, wherein the target model is obtained by training a first model based on the training method of the feature extraction model described above; and performing scene retrieval based on the feature extraction result to recommend graphic codes based on the scene retrieval result.

[0007] According to a third aspect of this disclosure, a training apparatus for a feature extraction model is provided, comprising: a first training module for training a first model to be trained based on a first training image set, the first model including a cascaded first feature extraction module and a second feature extraction module, the first feature extraction module including a cascaded point convolution module and a depth convolution module; a processing module for replacing the second feature extraction module in the trained first model with a feature mapping module to obtain a second model; and a second training module for training the second model using the first training image set, fusing the point convolution module and the depth convolution module of the first feature extraction module in the trained second model, and determining the fused second model as a target model.

[0008] According to a fourth aspect of this disclosure, an information recommendation apparatus is provided, comprising: a target detection module, configured to acquire a collected image and perform target detection on the collected image based on an object detection model to obtain a graphic code detection result; a feature extraction module, configured to, if it is determined from the graphic code detection result that the collected image includes a graphic code, perform feature extraction on the collected image based on a target model to obtain a feature extraction result, wherein the target model is obtained by training a first model based on the training method of the feature extraction model described above; and a recommendation module, configured to perform scene retrieval based on the feature extraction result, so as to recommend graphic codes based on the scene retrieval result.

[0009] According to a fifth aspect of this disclosure, an electronic device is provided, characterized in that it includes: a processor; and a memory for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to implement the method described above.

[0010] According to a sixth aspect of this disclosure, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method described above.

[0011] The training method for the feature extraction model provided in the embodiments of this disclosure, on the one hand, involves training a first model to be trained using a first training image set, then replacing the second feature extraction model in the trained first model with a feature mapping module to obtain a second model. The second model is then trained using the first image training set. The first feature extraction module is trained with the assistance of the second feature extraction module, enabling it to learn the distribution of images in the feature space. After replacing the second feature extraction module with the feature mapping module, the second model is trained again using the first image set, further enhancing the model's feature extraction capability and increasing its feature mapping capability, thereby further improving the model's performance. On the other hand, using the second model, which integrates point convolution and depth convolution modules, as the target model improves model accuracy while reducing model complexity. Furthermore, cascading the point convolution and depth convolution modules reduces model parameters and computational load, thereby improving the model's inference speed. Finally, when the high-accuracy, low-complexity target model obtained using this method is used for image feature extraction, it effectively improves the accuracy of feature extraction, which is of practical significance for further expanding application scenarios for information recommendation using features extracted by this method.

[0012] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0013] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. In the drawings:

[0014] Figure 1 A schematic diagram illustrating the stages involved in an information recommendation scheme applying an embodiment of the present disclosure is shown;

[0015] Figure 2 A flowchart illustrating a training method for a feature extraction model in an exemplary embodiment of the present disclosure is shown schematically.

[0016] Figure 3 This schematic diagram illustrates the network structure composition of a first model in an exemplary embodiment of the present disclosure.

[0017] Figure 4 The flowchart schematically illustrates an implementation method for obtaining a target model by training a second model in an exemplary embodiment of the present disclosure;

[0018] Figure 5This illustration schematically depicts a convolution merging process according to an exemplary embodiment of the present disclosure;

[0019] Figure 6 This schematically illustrates a flowchart of an exemplary embodiment of the present disclosure, in which the point convolutional module and the depth convolutional module of the first feature extraction module in the trained second model are fused, and the second model containing the fusion result is determined as the target model.

[0020] Figure 7 This schematically illustrates another convolutional merging diagram in an exemplary embodiment of the present disclosure;

[0021] Figure 8 This schematically illustrates yet another convolutional merging diagram in an exemplary embodiment of the present disclosure;

[0022] Figure 9 This schematically illustrates another convolutional merging diagram in an exemplary embodiment of the present disclosure;

[0023] Figure 10 The illustration schematically shows a 3×3 asymmetric convolution in an exemplary embodiment of the present disclosure;

[0024] Figure 11 This schematically illustrates a flowchart of an information recommendation method according to an exemplary embodiment of the present disclosure;

[0025] Figure 12 This schematic diagram illustrates the structure of the backbone network of an object detection model in an exemplary embodiment of the present disclosure.

[0026] Figure 13 This schematic diagram illustrates an example of setting an anchor point in an exemplary embodiment of the present disclosure.

[0027] Figure 14 This schematically illustrates a diagram of an anchor frame and a real frame in an exemplary embodiment of the present disclosure;

[0028] Figure 15 A flowchart illustrating an implementation of a compressed feature vector in an exemplary embodiment of the present disclosure is shown.

[0029] Figure 16 This schematically illustrates a method for partitioning a first feature vector according to an exemplary embodiment of the present disclosure;

[0030] Figure 17 This diagram illustrates an application scenario of an information recommendation method according to an exemplary embodiment of the present disclosure.

[0031] Figure 18 This illustration schematically depicts a process diagram of accurately recommending graphic codes to users in an application scenario according to an exemplary embodiment of the present disclosure;

[0032] Figure 19 This schematic diagram illustrates the composition of a training apparatus for a feature extraction model in an exemplary embodiment of the present disclosure.

[0033] Figure 20 This schematic diagram illustrates the composition of an information recommendation device according to an exemplary embodiment of the present disclosure;

[0034] Figure 21 A schematic diagram of an electronic device to which embodiments of the present disclosure may be applied is shown. Detailed Implementation

[0035] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0036] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0037] Figure 1 The diagram illustrates the stages involved in an information recommendation scheme to which embodiments of this disclosure can be applied. Embodiments of this disclosure relate to a model training stage and a model application stage.

[0038] During the model training phase, the initial model can be trained using the training image set until the model training completion conditions are met, such as until convergence, to obtain the trained target model. During the model application phase, the trained target model is used to extract features (object detection) from the acquired images to output the feature extraction results (detection results) corresponding to the acquired images.

[0039] In some embodiments of this disclosure, the model training and model application phases of this disclosure can be executed by a terminal device. That is, both the model training and model application schemes of this disclosure are implemented by the same terminal device. The terminal device may also be referred to as a mobile terminal, terminal, mobile device, etc. This disclosure does not limit the type of terminal device and may include, but is not limited to, smartphones, tablets, smart wearable devices, personal computers, etc. The server may also be referred to as the cloud or server-side, and it can be a standalone server or a cluster of multiple devices.

[0040] In other embodiments of this disclosure, the model training and model application phases of this disclosure can be performed by a server. Images used during model training and application can be uploaded to the server by the terminal device.

[0041] In some other embodiments of this disclosure, the model training phase of this disclosure can be performed by a server, which then sends the trained model to a terminal device, which then performs the model application phase of this disclosure.

[0042] QR code scanning technology is widely used in various application scenarios. By scanning a QR code, graphic code information issued according to the code issuance rules can be identified, and the corresponding business can be realized, such as receiving and making payments, opening websites, and downloading applications. These include, but are not limited to, QR codes, barcodes, and other graphic codes arranged according to certain encoding rules to express a set of information.

[0043] In many scenarios, the low efficiency and poor accuracy of QR code scanning affect the user's scanning experience. When scanning a QR code, feature extraction of the image is required. However, the models used for feature extraction in related technologies ignore the influence of image interference factors on model training, resulting in low feature extraction accuracy after training. This affects the success rate of scanning to a certain extent. Furthermore, when there are too many image interference factors that prevent successful scanning, it increases the user's waiting time and brings a bad user experience.

[0044] Based on one or more of the above-mentioned problems, this example implementation first provides a method for training a feature extraction model. (Reference) Figure 2 As shown, the training method for this feature extraction model may include the following steps S210 to S230:

[0045] In step S210, a first model to be trained is trained based on a first training image set, wherein the first model includes a cascaded first feature extraction module and a second feature extraction module, and the first feature extraction module includes a cascaded point convolution module and a depth convolution module.

[0046] In an exemplary embodiment of this disclosure, the first training image set includes images and image labels corresponding to different scene types, such as images collected in a payment scenario, images collected in a website opening scenario, images collected in an application download scenario, and so on.

[0047] In some possible embodiments, the images in the first training image set can be enhanced images obtained by preprocessing the initial images. For example, if the initial image is overexposed or underexposed, normalization can be used to transform each pixel value.

[0048] For example, an initial image I of size M×N that is overexposed or underexposed can be normalized using the following method.

[0049] First, obtain the average gray value of the initial image I using formula (1):

[0050]

[0051] Where I(c,j,k) is the pixel value of the c-th channel at position (j,k) in the initial image, 0≦j <M,0≦k<N,0≦c<3。

[0052] When u is less than the first preset value, it indicates that the initial image I is too dark; when u is greater than the second preset value, it indicates that the image I is overexposed. Each pixel of the initial image I can be transformed using formula (2):

[0053]

[0054] Where max(I(c)) is the maximum pixel value of the c-th channel of the initial image I; min(I(c)) represents the minimum pixel value of the c-th channel of the initial image I. If u is in the interval [first preset value, second preset value], then let I... p =I, meaning the pixel value remains unchanged.

[0055] It is worth noting that the first preset value and the second preset value can be set according to actual needs, and there are no special restrictions on them. In addition, other preprocessing methods can be used to perform image enhancement processing on the initial image to form images in the first training image set based on the preprocessed initial image. The embodiments of this disclosure include, but are not limited to, the above image preprocessing methods.

[0056] By enhancing images that are too dark or too exposed, the model can adapt to application scenarios that are too dark or too exposed.

[0057] The first model to be trained can be a deep neural network, including a cascaded first feature extraction module and a second feature extraction module, used to extract features from the image. The first feature extraction module may include cascaded point convolution modules and depthwise convolution modules. There are no specific restrictions on the network structure of the second feature extraction module. For example, the second feature extraction module may include operations such as convolution calculation, non-linear activation function calculation, and pooling calculation. For instance, the second feature extraction module can be a global average pooling network structure. The first feature extraction module differs from DSC (Depth Separable Convolution). In this embodiment, point convolution modules are used first, followed by depthwise convolution modules, which reduces computational complexity.

[0058] The first model may include a first feature extraction module with multiple layers having the same network structure. A pre-feature extraction module may also be included before the first feature extraction module. The embodiments of this disclosure do not impose special restrictions on the specific network structure of the pre-feature extraction module. For example, the pre-feature extraction module may include operations such as convolution calculation and pooling calculation.

[0059] For example, Figure 3 The diagram illustrates the network structure of a first model according to an embodiment of this disclosure, wherein s1 and s2 in the first model represent "step size 1" and "step size 2", respectively. Of course, Figure 3 This is merely an example; the number of first feature extraction modules in the first model can be adjusted according to actual needs in the embodiments disclosed herein.

[0060] Training the first model based on the first training image set involves inputting images corresponding to different scene types into the first model to obtain image classification results. Then, based on the image classification results and image labels, a loss function is determined, and the parameters of the first model are adjusted using the loss function to obtain the trained first model. In other words, the first model is initially trained using the first training image set, and the trained first model is a classification model.

[0061] In step S220, the second feature extraction module in the trained first model is replaced by the feature mapping module to obtain the second model.

[0062] In an exemplary embodiment of this disclosure, after training the first model with the first training image, the second feature extraction module can be replaced with a feature mapping module. The feature mapping module is used to map the image features containing spatial structure information extracted by the first feature extraction module to a set dimension, so as to obtain the feature weights corresponding to the image features from the set dimension. The feature weights reflect the effectiveness of the image features. The larger the feature weights, the higher the effectiveness of the image features, that is, the more important they are. Conversely, the smaller the feature weights, the lower the effectiveness of the image features.

[0063] Therefore, after the feature mapping module maps image features containing spatial structure to a set dimension, the image features can be further filtered based on the obtained feature weights.

[0064] The feature mapping module can be a 1*1 convolution. If the first feature extraction module outputs a feature map of size M×M, it includes M... 2 The feature map is generated from 3 eigenvectors. After passing through the feature mapping module, the output is an M×M matrix (called the scoring matrix S), where each element corresponds to a different eigenvector. In the scoring matrix S... ij The larger the value of , the greater the feature weight of the corresponding feature vector, and the higher the effectiveness of the feature vector. Therefore, based on the feature weight, we can start from M 2 Selecting the target feature vector from the feature vectors. For example, M can be sorted according to the magnitude of the feature weights. 2 The feature vectors are sorted, and the target feature vectors with higher feature weights are selected from the sorted results. The number of targets is not specifically limited, but can be determined according to the actual application requirements.

[0065] It is worth noting that the feature mapping module can also be a weight matrix trained according to actual needs. The embodiments of this disclosure include, but are not limited to, the feature mapping module described above.

[0066] In this embodiment, the first feature extraction module is first trained with the assistance of the second feature extraction module, so that the first feature extraction module learns the distribution of the image in the feature space. Then, the network layer parameters of the first feature extraction module are fixed, and the second feature extraction module in the trained first model is replaced by the feature mapping module. This allows the trained second model to obtain the distribution of the image in the feature space and perform feature filtering based on feature weights, which helps to remove invalid features introduced by image interference factors and improve the accuracy of feature extraction.

[0067] In step S230, the second model is trained using the first training image set, and the point convolution module and depth convolution module of the first feature extraction module in the trained second model are fused together, and the fused second model is determined as the target model.

[0068] In an exemplary embodiment of this disclosure, a second model can be trained using a first training image set to obtain a trained second model. To ensure the target model has high inference speed during application, the point convolutional module and depthwise convolutional module of the first feature extraction module in the trained second model can be merged into a single convolutional module, reducing the number of model parameters. For example, a reparameterization method can be used for convolution merging.

[0069] The training method for the feature extraction model in this embodiment includes, on the one hand, training a first model to be trained using a first training image set, replacing the second feature extraction model in the trained first model with a feature mapping module to obtain a second model, and then training the second model using the first training set. After training the first feature extraction module with the assistance of the second feature extraction module, enabling the first feature extraction module to learn the distribution of images in the feature space, replacing the second feature extraction module with the feature mapping module, and then training the second model again using the first image set, the model's feature extraction capability and feature mapping capability are further improved, thereby further enhancing the model's performance. On the other hand, using the second model, which integrates point convolution modules and depth convolution modules, as the target model improves model accuracy while reducing model complexity. Furthermore, cascading the point convolution modules and depth convolution modules reduces model parameters and computational load, thereby improving the model's inference speed.

[0070] In an exemplary embodiment, a method is provided for obtaining a target model by training a second model. Training the second model using a first training image set, fusing the point convolutional module and the depth convolutional module of the first feature extraction module in the trained second model, and determining the fused second model as the target model may include steps S410 to S440:

[0071] Step S410: Use the first feature extraction module in the second model to extract features from the images in the first training image set to obtain a feature map containing multiple feature vectors.

[0072] In this process, the first feature extraction module (a cascaded point convolution module and a depth convolution module) in the second model can be used to extract features from the images in the first training image set, resulting in a feature map containing multiple feature vectors.

[0073] Step S420: Use the feature mapping module in the second model to perform feature mapping processing on the feature map, obtain the feature weights corresponding to each feature vector, and determine the target feature vector from multiple feature vectors based on the feature weights.

[0074] This embodiment of the disclosure takes a 1*1 convolution as an example to illustrate the feature mapping module. The feature map is mapped using a 1*1 convolution to obtain a scoring matrix corresponding to the feature map. Each matrix element corresponds to a different feature vector. The value of the matrix element is used to reflect the feature weight of the corresponding feature vector. Multiple feature vectors of the feature map are filtered according to the feature weight to determine the target feature vector.

[0075] In step S430, the parameters of the second model are adjusted based on the target feature vector to obtain the trained second model.

[0076] After obtaining the target feature vector, the error can be determined based on the feature vectors of each image sample in the first training image set. The parameters of the second model can then be adjusted according to the error to obtain the trained second model.

[0077] Step S440: Fuse the point convolution module and the depth convolution module of the first feature extraction module in the trained second model, and determine the fused second model as the target model.

[0078] Based on the network structure of the first feature extraction module, the point convolutional module and the depth convolutional model of the first feature extraction module can be fused, and the second model that fuses the point convolutional module and the depth convolutional module can be used as the target model. Since convolution merging is performed in the first feature extraction module, the number of parameters can be reduced, making the model lightweight, improving the inference speed, and also facilitating the deployment of the trained model in scenarios with low power consumption requirements.

[0079] In this embodiment, the first feature extraction module includes a point convolution module comprising multiple parallel first point convolution modules, each first point convolution module comprising a cascaded first convolutional layer and a batch normalization layer; the depth convolution module includes multiple parallel first depth convolution modules, each first depth convolution module comprising a cascaded second convolutional layer and a batch normalization layer. For specific structures, please refer to [reference needed]. Figure 5 As shown. To achieve the merging of convolutional layers and batch normalization layers, the convolution within the dashed box can be an additional convolution added during module merging, such as by adding a Dirac delta function (adding convolution) and merging with the batch normalization layer. Of course, the convolution within the dashed box can also be the same as the first convolutional layer; the choice depends on the specific situation.

[0080] Based on the structure of the first feature extraction module, such as Figure 6 As shown, fusing the point convolutional module and the depth convolutional module of the first feature extraction module in the trained second model, and determining the second model containing the fusion result as the target model may include steps S610 to S650:

[0081] Step S610: Merge the first convolutional layer and batch normalization layer with a cascaded relationship in the point convolutional module of the trained second model to obtain multiple parallel second point convolutional modules.

[0082] For each convolutional kernel of the first convolutional layer, the model learning parameters of the corresponding batch normalization layer can be used to reparameterize each convolutional kernel and its bias, thereby reparameterizing the first convolutional layer and the batch normalization layer, which have a cascade relationship, into a single convolutional module.

[0083] The first convolutional layer and its corresponding batch normalization layer can be calculated using formula (3) as follows:

[0084]

[0085] Among them, O j Let I be the j-th channel of the output of the first convolutional layer, I be the input image, F be the convolutional layer, * be the convolution symbol, and μ be the input image. j and σ j These are the pixel mean and pixel standard deviation of the j-th channel of the input image I, respectively, γ j and β j The parameters for the batch normalization layer are learned during the training of the second model. The bias of the convolution sum is set to 0.

[0086] Combining formula (3) with the additivity of convolution operations, each convolution kernel (each channel parameter) of the first convolutional layer is reparameterized as follows:

[0087]

[0088] And, the bias b corresponding to each convolution kernel of the first convolutional layer j Reparameterization is as follows:

[0089]

[0090] By reparameterizing the first convolutional layer and the batch normalization layer using formulas (3) to (5), the convolutional layer and the batch normalization layer with a cascaded relationship are merged into a second point convolutional module.

[0091] It should be noted that if the first feature extraction module has multiple parallel point convolution modules, each point convolution module can be convolved and merged into a single convolution module. This will not be elaborated further.

[0092] Step S620: Merge the second convolutional layers and batch normalization layers that have a cascaded relationship in the depthwise convolutional modules to obtain multiple parallel second depthwise convolutional modules.

[0093] The second convolutional layer and the batch normalization layer in the deep convolutional module can be merged into a second deep convolutional module. For each convolutional kernel of the second convolutional layer, the model learning parameters of the corresponding batch normalization layer are used to reparameterize each convolutional kernel and the bias of each convolutional kernel.

[0094] For a detailed description of the specific merging method, please refer to step S610, which describes merging the first convolutional layer in the point convolution module with the corresponding batch normalization layer. This will not be repeated here.

[0095] It should be noted that the first feature extraction module includes multiple parallel point convolutional modules. The first convolutional layer in each point convolutional module is merged with its corresponding batch normalization layer to obtain multiple parallel second point convolutional modules. Similarly, the first feature extraction module includes multiple parallel depthwise convolutional modules. The second convolutional layer in each depthwise convolutional module is merged with its corresponding batch normalization layer to obtain multiple parallel second depthwise convolutional modules. A schematic diagram illustrating the fusion operation of convolutional layers and batch normalization layers with a cascaded relationship can be found in [reference needed]. Figure 7 As shown.

[0096] Step S630: Merge multiple parallel second-point convolutional modules to obtain a third-point convolutional module, and merge multiple parallel second-depth convolutional modules to obtain a third-depth convolutional module.

[0097] Multiple parallel second-point convolutional modules can be merged into a single third-point convolutional module. When multiple convolutional layers have the same input and hyperparameters (such as kernel size and stride), if the outputs of each second-point convolutional module are directly added, a new convolutional kernel can be formed by fusing the parameters of the kernels of multiple parallel third-point convolutional modules. Similarly, a new convolutional kernel can be formed by fusing the parameters of the kernels of multiple parallel third-depth convolutional modules. This allows for the merging of multiple parallel second-point convolutional modules to obtain a third-point convolutional module, and the merging of multiple parallel second-depth convolutional modules to obtain a third-depth convolutional module. A schematic diagram illustrating the merging of multiple parallel convolutional modules can be found here. Figure 8 As shown.

[0098] It's worth noting that when merging multiple parallel convolutional modules, if at least two of them have kernels with different dimensions, the module with the smaller kernel dimension can be padded to the dimension of the module with the larger kernel dimension to make the kernel dimensions of all parallel modules the same. For example, padding with zeros is acceptable, and there are no special restrictions on this.

[0099] Step S640: By reparameterizing the third point convolutional model and the third depth convolutional module, the third point convolutional module and the third depth convolutional module are merged into the target convolutional module.

[0100] The third-point convolutional module and the third-depth convolutional module can be merged into a single target convolutional module using a reparameterization operation. The convolutional kernels of the third-point and third-depth convolutional modules may have different dimensions; in such cases, different reparameterization methods are used for merging, and no special requirements are imposed on this.

[0101] In some possible embodiments, if the dimension corresponding to the convolution kernel of the third point convolution module is smaller than the dimension corresponding to the convolution kernel of the third depth convolution module, the transpose result of the convolution kernel of the third point convolution module can be obtained, and the transpose result can be merged with the convolution kernel corresponding to the third depth convolution module to obtain the target convolution module.

[0102] For example, if the third point convolution module is a 1×1 convolution (kernel denoted as F1) and the third depth convolution module is a 3×3 convolution (kernel denoted as F2), when F1 and F2 are concatenated as input image I, F1 and F2 are reparameterized into a new convolution using formula (6):

[0103] F = F2 * F1 T (6)

[0104] Among them, F1 T This is the transpose of convolution kernel F1. For merging the third-point convolutional module with the third-depth convolutional module, see [link to documentation]. Figure 9 As shown.

[0105] By merging the point convolution module and the depth convolution module in the first feature extraction module into a single convolution (target convolution module) through steps S610 to S640, the number of model parameters is reduced, which is beneficial to improving the running speed of the model and reducing the model deployment cost.

[0106] Step S650: The second model containing the target convolutional module is identified as the target model.

[0107] After merging the point convolution module and the depth convolution module in the first feature extraction module into a single target convolution module, the target model is determined based on the target convolution module in the first feature extraction module and the second feature extraction module.

[0108] In one exemplary embodiment, the second convolutional layer in the depthwise convolutional module is a grouped convolutional layer, and the depthwise convolutional module contains at least one asymmetric grouped convolutional layer. Grouped convolution and asymmetric convolution can further reduce the number of parameters.

[0109] Taking asymmetric convolution as an example, asymmetric convolution is learnable only in certain regions of the image, such as... Figure 10 A schematic diagram of a 3×3 asymmetric convolution according to an exemplary embodiment of the present disclosure is shown. The identifier K corresponds to learnable parameters, and the blank portion is always 0. Asymmetric convolution allows for the extraction of image features using fewer convolution parameters.

[0110] The following describes the model application stage of embodiments of this disclosure. Embodiments of this disclosure also provide an information recommendation method applying the trained target model described above, such as... Figure 11As shown, the information recommendation method may include the following steps S1110 to S1130:

[0111] Step S1110: Acquire the captured image and perform target detection on the captured image based on the object detection model to obtain the graphic code detection result.

[0112] In an exemplary embodiment of this disclosure, the acquired image is an image captured by a terminal device, such as an image captured during a barcode scanning operation. After obtaining the acquired image, object detection can be performed on the acquired image based on an object detection model to determine whether a graphic barcode exists. The object detection model is an object detection model obtained by training a third model to be trained on a second training image set. The second training image set includes acquired image samples and their corresponding labels.

[0113] Step S1120: If the image contains a graphic code based on the graphic code detection result, feature extraction is performed on the image based on the target model to obtain the feature extraction result. The target model is obtained by training the first model based on the above feature extraction model training method.

[0114] After determining that the acquired image contains a graphic code based on the graphic code detection result of the object detection model, the graphic code can be feature extracted, and the scene image can be retrieved based on the feature extraction result.

[0115] It should be noted that the training method of the target model has been described in detail in the above-described embodiment of the training method of the feature extraction model, and will not be repeated here.

[0116] Step S1130: Perform scene retrieval based on feature extraction results, and recommend graphic codes based on scene retrieval results.

[0117] After obtaining the feature extraction results of the acquired images, scene retrieval can be performed based on the feature extraction results, and graphic codes can be recommended to users based on the scene retrieval results.

[0118] The information recommendation method of this disclosure can perform target detection and feature extraction processing on the collected image, and actively recommend graphic codes to the user based on the processing results. When the user cannot accurately scan the code at a distance, the recommended graphic code can be used to quickly complete the scanning operation, which greatly improves scanning efficiency, saves scanning time, and enhances the user experience.

[0119] In one exemplary embodiment, an implementation method for training an object detection model is provided. Before performing object detection on the acquired images based on the object detection model to obtain the graphic code detection result, a third model to be trained can be trained based on a second training image set to obtain the object detection model.

[0120] The object detection model includes cascaded third and fourth feature extraction modules. The third feature extraction module comprises cascaded point convolutional and depthwise convolutional modules. In other words, the object detection model can use the same backbone network structure as the feature extraction model. For example... Figure 12 A schematic diagram of the backbone network of an object detection model according to an exemplary embodiment of the present disclosure is shown, as follows: Figure 12 As shown, the number of third feature extraction modules in the object detection model can be multiple, and this embodiment does not impose any special restrictions on the network structure of the fourth feature extraction module.

[0121] One approach is to use anchor points for model training. Training the third model to be trained based on the second training image set to obtain the object detection model can include:

[0122] During the training of the third model to be trained, anchor points are set in the feature map output by the target third feature extraction module and the training process continues.

[0123] The point convolution module and the depth convolution module of the third feature extraction module in the trained third model are fused together, and the fused third model is determined as the object detection model.

[0124] See also Figure 12 As shown, in this embodiment, multiple anchor points can be set at each pixel position in the feature maps output by Conv_4, Conv_6, and Conv_8, such as setting three anchor points with aspect ratios of {0.9, 1.0, 1.1} respectively. This embodiment allows selection of the number of anchor points and their corresponding aspect ratios according to actual needs, without any special limitations.

[0125] Setting anchor points involves determining a series of candidate bounding boxes before object detection, with objects in the image being covered by these predefined boxes by default. Each pixel in the final output image has a set of anchor boxes (e.g., three anchor boxes). The model's output consists of the offsets and classification information for all anchor boxes. For example, an anchor box containing only the background is classified as "background" with an offset of 0, while an anchor box containing an object is classified as the object's category with an offset equal to the offset between the anchor box and the object's ground truth bounding box.

[0126] like Figure 13 A schematic diagram illustrating an exemplary embodiment of the present disclosure for setting an anchor point is shown. Figure 13 One anchor point corresponds to three different aspect ratios for the anchor frame. See also Figure 14 As shown, the solid-lined boxes represent anchor boxes, and the dashed-lined boxes represent truth boxes. The position of the anchor box can be determined by defining its top-left vertex, width, and height, and its offset relative to the truth box. Setting anchor points during training can reduce computational load and improve training efficiency.

[0127] In one exemplary embodiment, a method for performing scene retrieval based on feature extraction results and recommending graphic codes based on scene retrieval results is also provided, which may include:

[0128] The system utilizes an object detection model to extract features from scene images within a predefined scene set, obtaining feature vector sets corresponding to each scene image. The extracted features are then matched against the feature vector sets corresponding to each scene image in the predefined scene set, and graphic codes are recommended based on the matching results. In other words, the system first obtains the feature extraction results from the acquired images, then acquires the features (feature vector sets) of each scene image from the predefined scene set, and finally determines the target scene image through feature matching to recommend the graphic code corresponding to the target scene image.

[0129] In some possible embodiments, to further improve scene retrieval efficiency, a method for compressing feature vectors is also provided, which may include steps S1510 to S1540:

[0130] Step S1510: Divide the first feature vector in the feature vector set corresponding to the scene image into multiple first sub-vectors.

[0131] For scene images in the preset scene set, features can be extracted using the target model trained above, resulting in a feature vector set for each scene image. For example, each scene image includes 20 feature vectors, but the number of feature vectors in each scene image can be adjusted according to actual needs.

[0132] Each first feature vector can be segmented to obtain multiple first sub-vectors of length [length missing]. For example... Figure 16 A schematic diagram illustrating a method for partitioning a first feature vector according to an exemplary embodiment of the present disclosure is shown. If the first feature vector is 128-dimensional, it is divided into 8 first sub-vectors, each of which has a length of 16.

[0133] Step S1520: Perform clustering processing on the first sub-vector corresponding to the scene image to obtain N cluster centers and the index values ​​of the cluster centers, where N is a positive integer.

[0134] For each scene image, cluster all the first sub-vectors corresponding to that scene image to obtain N cluster centers, and determine the index value (e.g., 1, 2, 3, ..., N) of each cluster center. The clustering algorithm can be K-means, K-center algorithm, etc., and no special requirements are made.

[0135] Step S1530: Obtain the nearest target cluster center for each first sub-vector, and use the index value of the target cluster center as the index value of the first sub-vector.

[0136] For each scene image, from the N cluster centers, obtain the nearest target cluster center for each first sub-vector corresponding to the scene image, and replace the first sub-vector with the target cluster center.

[0137] For example, if the first sub-vector (x1, x2, x3, ... xm) of the scene image is L closest to the target cluster center, and the index value of the target cluster center L among the N cluster centers is 8, then the first sub-vector (x1, x2, x3, ... xm) is replaced with 8, and so on, to obtain the index value of each first sub-vector of the scene image.

[0138] Step S1540: For each first feature vector, determine the feature representation of the first feature vector based on the index values ​​of each first sub-vector of the first feature vector.

[0139] For each first feature vector of a scene image, after obtaining the index values ​​of each first sub-vector, each first feature vector can include multiple index values ​​(each corresponding to a first sub-vector). Based on this, any scene image can include multiple first feature vectors, and each first feature vector includes multiple index values. See details... Figure 16 As shown.

[0140] This embodiment of the disclosure can compress the first feature vector of scene images in a preset scene set into an index value to improve the processing speed of subsequent scene retrieval and achieve near real-time retrieval.

[0141] In an exemplary embodiment, the feature extraction result of the acquired image includes multiple second feature vectors. Matching the feature extraction result with the feature vector set corresponding to each scene image in a preset scene set, and recommending graphic codes based on the matching result, may include:

[0142] First, each second feature vector is divided into multiple second sub-vectors. Second, for any second feature vector, the distance value between each second sub-vector and each cluster center is obtained, and the distance matrix is ​​determined based on the distance values ​​corresponding to all the second sub-vectors. Finally, based on the feature representation of the first feature vector and the distance matrix corresponding to the second feature vector of each scene image, the feature extraction result is matched with the feature vector set, and the image code is recommended based on the matching result.

[0143] The number of cluster centers is the same as the number of cluster centers determined when processing scene images in the preset scene set. When matching the acquired image with scene image A in the preset scene set, the cluster center corresponding to scene image A is used. Similarly, when matching the acquired image with scene image B in the preset scene set, the cluster center corresponding to scene image B is used, and so on.

[0144] The following example illustrates the matching of a captured image with scene image A in a preset scene set. For instance, scene image A includes 20 first feature vectors, and the number of corresponding cluster centers is k. The captured image also includes 20 second feature vectors (both extracted through the target model).

[0145] First, any one of the 20 second feature vectors (128 dimensions) of the acquired image is divided into 8 second sub-vectors vi, denoted as v1, v2, v3, ..., v8 respectively.

[0146] Secondly, for any second feature vector, the distance between each second sub-vector vi and each cluster center is obtained, resulting in 8 distance values, such as {d11, d12, d13…d18}. A distance matrix, i.e., an 8×k matrix, is then determined based on the 8 distance values ​​corresponding to all the second sub-vectors of the second feature vector, where k is the number of cluster centers. Each matrix element corresponds to a distance value for a different second sub-vector.

[0147] Finally, for any first feature vector (e.g., (1,2,8,6,9,11,26,23)) of any scene image in the preset scene set, the distance matrix corresponding to the second feature vector at the division position is queried according to the index value of the first sub-vector (e.g., 1). The distance value corresponding to each first sub-vector is obtained, and the target distance value of the first feature vector is obtained by summing the distance values ​​corresponding to all the first sub-vectors of the first feature vector. In other words, based on each second feature vector of the acquired image, the target distance value of each first feature vector in the scene image corresponding to the division position of the second feature vector is obtained. By analogy, the target distance values ​​of each first feature vector of the scene image can be obtained, so as to determine whether the scene image matches the acquired image based on the target distance values ​​of each first feature vector.

[0148] Specifically, the scene image corresponding to the target distance value of the first feature vector that meets a preset condition can be identified as the matching result, and the corresponding graphic code of the matching result can be recommended. For example, the scene image corresponding to the target distance value of the first feature vector that is less than a preset distance value (such as 0.2) can be identified as the target scene image.

[0149] By using the compressed scene image and the acquired image for feature matching, the matching efficiency can be improved, thereby improving the scene retrieval efficiency.

[0150] In one exemplary embodiment, an implementation method for recommending graphic codes based on matching results is also provided. Recommending graphic codes based on matching results may include:

[0151] The system determines the location information corresponding to the captured image, obtains the text recognition results of the captured image, and combines the matching results, location information, and text recognition results to recommend graphic codes.

[0152] Optionally, location information corresponding to the captured image can be obtained, such as the location information of the terminal performing the scanning operation. If the location corresponding to the retrieved target scene image meets the set conditions with the location information corresponding to the captured image, such as the location distance being within a specified range, then the graphic code corresponding to the target scene image can be recommended.

[0153] Optionally, the text recognition results of the acquired images can also be combined. For example, OCR (Optical Character Recognition) can be used to obtain text recognition results of the acquired images. If the text recognition results also include target text, such as "venue code", "health code" and "epidemic prevention code", the graphic code corresponding to the target scene image can be recommended.

[0154] By combining scene retrieval results, location information, and text recognition results, the similarity between the current scanning scene and scene images in a preset scene set is determined, thereby improving the accuracy of image code recommendation.

[0155] In an exemplary embodiment, candidate images can also be generated based on the acquired images, and graphic code recognition can be performed on the acquired images and candidate images. If the graphic code recognition result is incorrect, feature extraction can be performed on the acquired images based on the target model to obtain the feature extraction result, and the subsequent scene retrieval process can continue.

[0156] The generation of candidate images can include random processing of the acquired image to obtain at least one candidate image obtained through image enhancement. Image enhancement processing can yield multiple candidate images with unchanged information corresponding to the acquired image. Image enhancement processing methods include, but are not limited to, image scaling, image sharpening, image normalization, image binarization, image rotation, and image flipping, which can be selected according to actual needs.

[0157] Optionally, the acquired image I can be scaled to different degrees to obtain candidate images. Optionally, the acquired image I can first be Gaussian blurred using a Gaussian kernel to obtain Gaussian(I), and then processed using the following formula (7):

[0158] I s =I + 1.5 × (I - Gaussian(I)) (7)

[0159] Where Is is the sharpened candidate image.

[0160] Optionally, candidate images can be obtained by performing image standardization processing using "Max-min normalization" and "histogram equalization" on the acquired images respectively. Optionally, different binarization methods can also be used to binarize the acquired images, including but not limited to Gaussian mean binarization, adaptive mean binarization, and adaptive Otsu's method (OTSU) binarization, to obtain candidate images.

[0161] Optionally, the acquired images can be rotated and flipped to increase the diversity of candidate images and prevent computational redundancy caused by excessively high image similarity. For example, if 12 candidate images are obtained and numbered from 1 to 12, I1, I2, ..., I12, for candidate image I... j If (j mod 4) = 1, the candidate image is rotated 90 degrees clockwise (mod is the remainder operator); if (j mod 4) = 2, the candidate image is rotated 180 degrees clockwise and flipped; if (j mod 4) = 3, the candidate image is rotated 270 degrees clockwise and flipped; if (j mod 4) = 0, the candidate image remains unchanged.

[0162] Optionally, the candidate image can also be flipped in black and white pixels to handle color-flipped graphic codes. For example, for candidate image I... j If I j It is obtained by binarizing the Gaussian mean, then I j In the middle, pixels 0 and 1 are flipped: pixel 0 becomes 1; pixel 1 becomes 0, resulting in a new candidate image I. inv I j and I inv All were treated as candidate images for identification.

[0163] Furthermore, the kernel size can be adaptively adjusted whenever a kernel is required, thereby enriching the diversity of candidate images.

[0164] It should be noted that during the process of obtaining candidate images for graphic code recognition using the above image enhancement methods, the image enhancement method can be selected according to actual needs, or the image enhancement can be performed sequentially. Each time, at least one enhancement method can be used. As long as any candidate image successfully recognizes the graphic code, the subsequent image enhancement will stop, and the graphic code recognition result will be returned to the user.

[0165] In one possible embodiment, if the image code recognition result is correct, the preset scene set is updated based on the acquired image and the feature extraction result of the acquired image, that is, scene registration is performed.

[0166] If the image code recognition result is correct and the current scene image is not located in the preset scene set, the image code and its corresponding feature extraction result can be updated to the preset scene set. For example, a scene identifier can be randomly generated, and a record with the following format can be added to the preset scene set: (scene identifier, feature vector vi, number of correct matches), where i is a positive integer, and the number of correct matches is used for subsequent updates to the preset scene set. Correspondingly, a record with the following format can also be added to the scene image code library: (scene identifier, image code content). The image code content can be saved as a string, without special restrictions.

[0167] When the current scene image is not in the preset scene set, the target scene image corresponding to the current scene image in the preset scene set is obtained, and the record corresponding to the target scene image is updated according to the feature extraction results of the acquired image.

[0168] For example, if the number of records corresponding to the target scene image is less than a set threshold, then a record in the following format (scene identifier, feature vector vi, number of correct matches) is added to the target scene image; conversely, if the number of records corresponding to the target scene image is greater than the set threshold, then records are deleted based on the number of correct matches, and a record in the following format (scene identifier, feature vector vi, number of correct matches) is added.

[0169] It's worth noting that the number of correct matches in each record indicates how many times that record has been correctly matched; the higher the number, the more likely the record is to be recommended. Based on this, scene images in a preset scene set can be filtered according to the number of correct matches corresponding to those scene images, ensuring that the scene images in the preset scene set always have a high recommendation probability, thereby improving the accuracy of image code recommendations.

[0170] Figure 17 The diagram illustrates an application scenario of the information recommendation method according to an exemplary embodiment of the present disclosure. In many scenarios, the scanning operation may be affected by long distances and numerous interference factors, which can impact scanning efficiency and accuracy. Figure 18 This diagram illustrates the process of accurately recommending graphic codes to users in a given application scenario. The following section will combine... Figure 17 and Figure 18 The information recommendation method of this disclosure will be described.

[0171] Step S1810: Trigger the scanning operation to obtain the captured image.

[0172] Step S1820: Perform target detection on the acquired image based on the object detection model to obtain the graphic code detection result.

[0173] Step S1830: Determine whether the graphic code detection result includes a graphic code.

[0174] If the image code detection result does not include an image code, the scanning process ends; if the image code detection result includes an image code, proceed to step S1840.

[0175] Step S1840: Perform graphic code recognition on the acquired image and the corresponding candidate image.

[0176] If the image code recognition result is incorrect, proceed to step S1850; if the image code recognition result is correct but incorrect, proceed to step S1860 to perform scene registration and jump to the successful scan result.

[0177] Step S1850: Extract features from the acquired image based on the target model to obtain the feature extraction results, and perform scene matching based on the feature extraction results to obtain the scene matching results.

[0178] If the scene matching is successful, proceed to step S1870. Based on the matching result, location information, and text recognition result, a graphic code recommendation is made. For example, a prompt message can be displayed on the user's graphical interface to inform the user that a graphic code is currently available for recommendation. In response to the user's trigger operation, directly jump to the interface where the code was successfully scanned. If the scene matching fails, the scanning process ends.

[0179] It should be noted that the training of the object detection model and the target model, as well as the detailed steps of each step, have been described in the above embodiments and will not be repeated here.

[0180] As can be seen from the above, the method of this embodiment trains the first model to be trained using the first training image set, then replaces the second feature extraction model in the trained first model with the feature mapping module to obtain the second model. The second model is then trained using the first training set. After the first feature extraction module learns the distribution of images in the feature space by assisting in training the first feature extraction module, the second feature extraction module is replaced with the feature mapping module, and the second model is trained again using the first image set. This further enhances the model's feature extraction capability and increases its feature mapping capability, thereby further improving the model's performance. Using the second model, which integrates point convolution and depth convolution modules, as the target model improves model accuracy while reducing model complexity. Furthermore, cascading the point convolution and depth convolution modules reduces model parameters and computational load, thereby improving the model's inference speed. In addition, when the high-accuracy, low-complexity target model obtained using this method is used for image feature extraction, it effectively improves the accuracy of feature extraction, which is of practical significance for further expanding the application scenarios of information recommendation using feature extraction through this method.

[0181] It should be noted that the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of this disclosure, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0182] Further reference Figure 19 As shown, an exemplary embodiment of this disclosure provides a training apparatus 1900 for a feature extraction model, including a first training module 1910, a processing module 1920, and a second training module 1930. Wherein:

[0183] The first training module 1910 is used to train the first model to be trained based on the first training image set. The first model includes a cascaded first feature extraction module and a second feature extraction module. The first feature extraction module includes a cascaded point convolution module and a depth convolution module.

[0184] Processing module 1920 is used to replace the second feature extraction module in the trained first model with the feature mapping module to obtain the second model;

[0185] The second training module 1930 is used to train the second model using the first training image set, fuse the point convolution module and the depth convolution module of the first feature extraction module in the trained second model, and determine the fused second model as the target model.

[0186] In an exemplary embodiment, the first training image set includes images corresponding to different scene types; the first training module 1910 is configured to:

[0187] The images corresponding to the different scene types are input into the first model to obtain the classification results of the images; a loss function is determined based on the classification results and image labels, and the parameters of the first model are adjusted using the loss function to obtain the trained first model.

[0188] In one exemplary embodiment, the second training module 1930 is configured as follows:

[0189] The first feature extraction module in the second model is used to extract features from the images in the first training image set to obtain a feature map containing multiple feature vectors. The feature mapping module in the second model is used to perform feature mapping processing on the feature map to obtain the feature weights corresponding to each feature vector, and a target feature vector is determined from the multiple feature vectors based on the feature weights. The parameters of the second model are adjusted based on the target feature vector to obtain the trained second model. The point convolution module and the depth convolution module of the first feature extraction module in the trained second model are fused, and the fused second model is determined as the target model.

[0190] In one exemplary embodiment, the point convolution module includes a plurality of parallel first point convolution modules, each of which includes a cascaded first convolutional layer and a batch normalization layer; the depth convolution module includes a plurality of parallel first depth convolution modules, each of which includes a cascaded second convolutional layer and a batch normalization layer.

[0191] The second training module 1930 is configured as follows:

[0192] The first convolutional layer and batch normalization layer with a cascaded relationship in the point convolutional module of the trained second model are fused to obtain multiple parallel second point convolutional modules; the second convolutional layer and batch normalization layer with a cascaded relationship in the depth convolutional module are fused to obtain multiple parallel second depth convolutional modules; the multiple parallel second point convolutional modules are merged to obtain a third point convolutional module, and the multiple parallel second depth convolutional modules are merged to obtain a third depth convolutional module; the third point convolutional module and the third depth convolutional module are fused into a target convolutional module through reparameterization operations; the second model containing the target convolutional module is determined as the target model.

[0193] In one exemplary embodiment, the second training module 1930 is configured as follows:

[0194] For each convolutional kernel of the convolutional layer, the model learning parameters of the corresponding batch normalization layer are used to reparameterize each convolutional kernel and the bias of each convolutional kernel, so as to reparameterize the convolutional layer and the batch normalization layer with cascade relationship into a convolutional module; wherein, the model learning parameters of the batch normalization layer are learned by training the second model.

[0195] In one exemplary embodiment, the second training module 1930 is configured to fuse the parameters of the convolution kernels of the plurality of parallel convolution modules.

[0196] In an exemplary embodiment, if the dimension corresponding to the convolution kernel of the third point convolution module is smaller than the dimension corresponding to the convolution kernel of the third depthwise convolution module; the second training module 1930 is configured as follows:

[0197] Obtain the transpose of the convolution kernel of the third point convolution module; merge the transpose with the corresponding convolution kernel of the third depth convolution module to obtain the target convolution module.

[0198] In an exemplary embodiment, the second convolutional layer in the depth convolution module is a grouped convolutional layer, and the depth convolution module contains at least one asymmetric grouped convolutional layer.

[0199] The specific details of each module in the above-mentioned device have been described in detail in the method section of the implementation. For any undisclosed details, please refer to the implementation content of the method section, and therefore will not be repeated here.

[0200] refer to Figure 20 As shown, an exemplary embodiment of this disclosure provides an information recommendation device 2000, including a target detection module 2010, a feature extraction module 2020, and a recommendation module 2030. Wherein:

[0201] The target detection module 2010 is used to acquire the collected image and perform target detection on the acquired image based on the object detection model to obtain the graphic code detection result;

[0202] The feature extraction module 2020 is used to extract features from the acquired image based on the target model if it is determined that the acquired image contains a graphic code according to the graphic code detection result, and to obtain the feature extraction result. The target model is obtained by training the first model based on the feature extraction model training method described above.

[0203] The recommendation module 2030 is used to perform scene retrieval based on the feature extraction results, so as to recommend graphic codes according to the scene retrieval results.

[0204] In one exemplary embodiment, the information recommendation device 2000 further includes:

[0205] The third training module is configured to train the third model to be trained based on the second training image set to obtain the object detection model; wherein, the object detection model includes a cascaded third feature extraction module and a fourth feature extraction module, and the third feature extraction module includes a cascaded point convolution module and a depth convolution module.

[0206] In one exemplary embodiment, the number of the third feature extraction modules is multiple; the third training module is configured as follows:

[0207] During the training process of the third model to be trained, anchor points are set in the feature map output by the target third feature extraction module and the training process continues; the point convolution module and depth convolution module of the third feature extraction module in the trained third model are fused, and the fused third model is determined as the object detection model.

[0208] In one exemplary embodiment, the recommendation module 2030 is configured to:

[0209] The object detection model is used to extract features from scene images in a preset scene set to obtain a feature vector set corresponding to the scene images; the feature extraction results are matched with the feature vector sets corresponding to each scene image in the preset scene set, and image code recommendations are made based on the matching results.

[0210] In one exemplary embodiment, the information recommendation device 2000 further includes:

[0211] The feature processing module is configured to divide a first feature vector in the feature vector set corresponding to the scene image into multiple first sub-vectors; perform clustering processing on the first sub-vectors corresponding to each scene image to obtain N cluster centers and the index values ​​of the cluster centers, where N is a positive integer; obtain the nearest target cluster center for each first sub-vector, and use the index value of the target cluster center as the index value of the first sub-vector; and for each first feature vector, determine the feature representation of the first feature vector based on the index values ​​of each first sub-vector of the first feature vector.

[0212] In one exemplary embodiment, the feature extraction result includes multiple second feature vectors; the recommendation module 2030 is configured to:

[0213] Each second feature vector is divided into multiple second sub-vectors; for any second feature vector, the distance value between each second sub-vector and each cluster center is obtained, and a distance matrix is ​​determined based on the distance values ​​corresponding to all the second sub-vectors; based on the feature representation of the first feature vector of each scene image and the distance matrix corresponding to the second feature vector, the feature extraction result is matched with the feature vector set, and a graphic code recommendation is performed based on the matching result;

[0214] In one exemplary embodiment, the recommendation module 2030 is configured to:

[0215] For any first feature vector, based on the index value of the first sub-vector in the feature representation, a query is performed from the distance matrix of the second feature vector corresponding to the partition position to obtain the distance value corresponding to the first sub-vector, and the target distance value of the first feature vector is determined based on the distance values ​​corresponding to all the first sub-vectors of the first feature vector; the scene image corresponding to the target distance value of the first feature vector satisfying the preset conditions is determined as the matching result; and the graphic code corresponding to the matching result is recommended.

[0216] In an exemplary embodiment, the recommendation module 2030 is configured to: determine the location information corresponding to the acquired image; obtain the text recognition result of the acquired image; and combine the matching result, the location information, and the text recognition result to perform graphic code recommendation.

[0217] In an exemplary embodiment, the feature extraction module 2020 is configured to: generate candidate images based on the acquired images; perform graphic code recognition on the acquired images and the candidate images; if the graphic code recognition result is incorrect, perform feature extraction on the acquired images based on the target model to obtain feature extraction results.

[0218] In one exemplary embodiment, the information recommendation device 2000 further includes:

[0219] The update module is configured to update the preset scene set based on the acquired image and the feature extraction result of the acquired image if the image code recognition result is correct.

[0220] In an exemplary embodiment, the scene images in the preset scene set include corresponding correct matching counts; the update module is further configured to filter the scene images in the preset scene set according to the correct matching counts corresponding to the scene images in the preset scene set.

[0221] The specific details of each module in the above-mentioned device have been described in detail in the method section of the implementation. For any undisclosed details, please refer to the implementation content of the method section, and therefore will not be repeated here.

[0222] Those skilled in the art will understand that various aspects of this disclosure can be implemented as a system, method, or program product. Therefore, various aspects of this disclosure can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, collectively referred to herein as a "circuit," "module," or "system."

[0223] An exemplary embodiment of this disclosure also provides an electronic device for the above-described method. This electronic device may be the aforementioned imaging device or a server. Generally, the electronic device includes at least a processor and a memory, the memory for storing executable instructions of the processor, and the processor configured to perform the above-described method by executing the executable instructions.

[0224] The following is based on Figure 21 Taking the mobile terminal 2100 as an example, the construction of the electronic device in this disclosure embodiment will be described by way of example. Those skilled in the art will understand that, apart from components specifically designed for mobile purposes, Figure 21 The structure shown can also be applied to fixed-type devices. In other embodiments, the mobile terminal 2100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The components illustrated can be implemented in hardware, software, or a combination of software and hardware. The interface connections between the components are only schematic and do not constitute a limitation on the structure of the mobile terminal 2100. In other embodiments, the mobile terminal may also adopt a similar design to... Figure 21 Different interface connection methods, or combinations of multiple interface connection methods.

[0225] like Figure 21 As shown, the mobile terminal 2100 may specifically include: a processor 2101, a memory 2102, a bus 2103, a mobile communication module 2104, an antenna 1, a wireless communication module 2105, an antenna 2, a display screen 2106, a camera module 2107, an audio module 2108, a power module 2109, a sensor module 2110, and a photovoltaic sensor module 2111.

[0226] The processor 2101 may include one or more processing units, such as an AP (Application Processor), a modem processor, a GPU (Graphics Processing Unit), an ISP (Image Signal Processor), a controller, an encoder, a decoder, a DSP (Digital Signal Processor), a baseband processor, and / or an NPU (Neural-Network Processing Unit).

[0227] An encoder encodes (i.e., compresses) images or videos to reduce data size for easier storage or transmission. A decoder decodes (i.e., decompresses) the encoded image or video data to restore the original image or video data. The mobile terminal 2100 may support one or more encoders and decoders.

[0228] The processor 2101 can be connected to the memory 2102 or other components via the bus 2103.

[0229] The memory 2102 can be used to store computer executable program code, which includes instructions. The processor 2101 executes various functional applications and data processing of the mobile terminal 2100 by running the instructions stored in the memory 2102. The memory 2102 can also store application data.

[0230] The communication function of mobile terminal 2100 can be implemented through mobile communication module 2104, antenna 1, wireless communication module 2105, antenna 2, modem processor, and baseband processor. Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Mobile communication module 2104 can provide 3G, 4G, 5G and other mobile communication solutions for mobile terminal 2100. Wireless communication module 2105 can provide wireless communication solutions such as wireless LAN, Bluetooth, and near-field communication for mobile terminal 2100.

[0231] The display screen 2106 is used to implement display functions, such as displaying the user interface, images, videos, etc., and displaying abnormal prompts. The camera module 2107 is used to implement shooting functions, such as capturing images and videos to acquire scene images. The audio module 2108 is used to implement audio functions, such as playing audio and capturing voice. The power module 2109 is used to implement power management functions, such as charging the battery, supplying power to the device, and monitoring battery status. The sensor module 2110 may include one or more sensors for implementing corresponding sensing and detection functions.

[0232] Furthermore, exemplary embodiments of this disclosure also provide a computer-readable storage medium storing a program product capable of implementing the methods described above. In some possible embodiments, various aspects of this disclosure may also be implemented as a program product including program code that, when run on a terminal device, causes the terminal device to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure.

[0233] It should be noted that the computer-readable medium disclosed herein may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0234] In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can transmit, propagate, or transfer a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wireline, optical fiber, RF, etc., or any suitable combination thereof.

[0235] Furthermore, program code for performing the operations of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0236] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

Claims

1. A training method for a feature extraction model, characterized in that, include: The first model to be trained is trained based on the first training image set. The first model includes a cascaded first feature extraction module and a second feature extraction module. The first feature extraction module includes a cascaded point convolution module and a depth convolution module. The second feature extraction module in the trained first model is replaced by the feature mapping module to obtain the second model; The second model is trained using the first training image set. The point convolution module and the depth convolution module of the first feature extraction module in the trained second model are fused together, and the fused second model is determined as the target model. The step of training the second model using the first training image set, fusing the point convolution module and the depth convolution module of the first feature extraction module in the trained second model, and determining the fused second model as the target model includes: The first feature extraction module in the second model is used to extract features from the images in the first training image set to obtain a feature map containing multiple feature vectors. The feature map is processed by the feature mapping module in the second model to obtain the feature weights corresponding to each feature vector, and the target feature vector is determined from the plurality of feature vectors based on the feature weights. The parameters of the second model are adjusted based on the target feature vector to obtain the trained second model; The point convolution module and the depth convolution module of the first feature extraction module in the trained second model are fused together, and the fused second model is determined as the target model.

2. The method according to claim 1, characterized in that, The first training image set includes images corresponding to different scene types; training the first model to be trained based on the first training image set includes: The images corresponding to the different scene types are input into the first model to obtain the classification results of the images; The loss function is determined based on the classification result and image label of the image, and the parameters of the first model are adjusted using the loss function to obtain the trained first model.

3. The method according to claim 1, characterized in that, The point convolution module includes multiple parallel first point convolution modules, each of which includes a cascaded first convolutional layer and a batch normalization layer; the depth convolution module includes multiple parallel first depth convolution modules, each of which includes a cascaded second convolutional layer and a batch normalization layer. The step of fusing the point convolutional module and the depth convolutional module of the first feature extraction module in the trained second model, and determining the second model containing the fusion result as the target model, includes: The first convolutional layer and batch normalization layer with a cascaded relationship in the point convolutional module of the trained second model are fused to obtain multiple parallel second point convolutional modules. The second convolutional layers and batch normalization layers that have a cascaded relationship in the depth convolutional module are fused together to obtain multiple parallel second depth convolutional modules; The multiple parallel second point convolutional modules are merged to obtain a third point convolutional module, and the multiple parallel second depth convolutional modules are merged to obtain a third depth convolutional module. By reparameterizing the third point convolution module and the third depth convolution module, the third point convolution module and the third depth convolution module are fused into a target convolution module. The second model containing the target convolutional module is determined as the target model.

4. The method according to claim 3, characterized in that, The process of fusing convolutional layers and batch normalized layers that have a cascaded relationship includes: For each convolutional kernel of the convolutional layer, the model learning parameters of the corresponding batch normalization layer are used to reparameterize each convolutional kernel and the bias of each convolutional kernel, so as to reparameterize the convolutional layer and batch normalization layer with cascade relationship into a convolutional module. The model learning parameters of the batch normalization layer are obtained by training the second model.

5. The method according to claim 3, characterized in that, The process of merging multiple parallel convolutional modules includes: The parameters of the convolution kernels of the multiple parallel convolution modules are fused.

6. The method according to claim 3, characterized in that, If the dimension corresponding to the convolution kernel of the third point convolution module is smaller than the dimension corresponding to the convolution kernel of the third depthwise convolution module; the step of fusing the third point convolution module and the third depthwise convolution module into a target convolution module through reparameterization operations includes: Obtain the transpose result of the convolution kernel of the third point convolution module; The transposed result is merged with the corresponding convolution kernel of the third depthwise convolution module to obtain the target convolution module.

7. The method according to any one of claims 3 to 6, characterized in that, The second convolutional layer in the depthwise convolutional module is a grouped convolutional layer, and there is at least one asymmetric grouped convolutional layer in the depthwise convolutional module.

8. An information recommendation method, characterized in that, include: Acquire images and perform target detection on the acquired images based on an object detection model to obtain graphic code detection results; If the image is determined to contain a graphic code based on the graphic code detection result, feature extraction is performed on the image based on the target model to obtain the feature extraction result. The target model is obtained by training the first model based on the training method of the feature extraction model according to any one of claims 1 to 7. Scene retrieval is performed based on the feature extraction results, and graphic code recommendations are made based on the scene retrieval results.

9. The method according to claim 8, characterized in that, Before performing target detection on the acquired image based on the object detection model to obtain the graphic code detection result, the method further includes: The object detection model is obtained by training the third model to be trained based on the second training image set; The object detection model includes a cascaded third feature extraction module and a fourth feature extraction module. The third feature extraction module includes a cascaded point convolution module and a depth convolution module.

10. The method according to claim 9, characterized in that, The number of the third feature extraction modules is multiple; the process of training the third model to be trained based on the second training image set to obtain the object detection model includes: During the training of the third model to be trained, anchor points are set in the feature map output by the target third feature extraction module and the training process continues. The point convolution module and the depth convolution module of the third feature extraction module in the trained third model are fused together, and the fused third model is determined as the object detection model.

11. The method according to claim 8, characterized in that, The step of performing scene retrieval based on the feature extraction results, and then recommending graphic codes based on the scene retrieval results, includes: The object detection model is used to extract features from scene images in a preset scene set to obtain a feature vector set corresponding to the scene images; The feature extraction results are matched with the feature vector sets corresponding to each scene image in the preset scene set, and graphic code recommendations are made based on the matching results.

12. The method according to claim 11, characterized in that, Before matching the feature extraction results with the feature vector sets corresponding to each scene image in the preset scene set, and recommending graphic codes based on the matching results, the method further includes: The first feature vector in the feature vector set corresponding to the scene image is divided into multiple first sub-vectors; Clustering is performed on the first sub-vector corresponding to the scene image to obtain N cluster centers and the index values ​​of the cluster centers, where N is a positive integer; Obtain the nearest target cluster center for each of the first sub-vectors, and use the index value of the target cluster center as the index value of the first sub-vector; For each of the first feature vectors, the feature representation of the first feature vector is determined based on the index values ​​of each of the first sub-vectors of the first feature vector.

13. The method according to claim 12, characterized in that, The feature extraction result includes multiple second feature vectors; the step of matching the feature extraction result with the feature vector set corresponding to each scene image in the preset scene set, and recommending image codes based on the matching result, includes: Each second feature vector is divided into multiple second sub-vectors; For any second feature vector, obtain the distance value between each second sub-vector and each cluster center, and determine the distance matrix based on the distance values ​​corresponding to all second sub-vectors; Based on the feature representation of the first feature vector of each scene image and the distance matrix corresponding to the second feature vector, the feature extraction result is matched with the feature vector set, and a graphic code recommendation is performed based on the matching result.

14. The method according to claim 13, characterized in that, The process of matching the feature extraction results with the feature vector set based on the feature representation of the first feature vector of each scene image and the distance matrix corresponding to the second feature vector, and recommending image codes based on the matching results, includes: For any first feature vector, based on the index value of the first sub-vector in the feature representation, a query is performed from the distance matrix of the second feature vector corresponding to the partition position to obtain the distance value corresponding to the first sub-vector, and the target distance value of the first feature vector is determined based on the distance values ​​corresponding to all the first sub-vectors of the first feature vector. The scene image corresponding to the target distance value of the first feature vector satisfying the preset condition is determined as the matching result; The graphic codes corresponding to the matching results will be recommended.

15. The method according to claim 11, characterized in that, The step of recommending graphic codes based on matching results also includes: Determine the location information corresponding to the acquired image; Obtain the text recognition results of the acquired image, and combine the matching results, the location information, and the text recognition results to recommend graphic codes.

16. The method according to any one of claims 8 to 15, characterized in that, The feature extraction based on the target model of the acquired image is used to obtain the feature extraction result, including: Candidate images are generated based on the acquired images; The acquired image and the candidate image are subjected to graphic code recognition; If the image code recognition result is incorrect, feature extraction is performed on the acquired image based on the target model to obtain the feature extraction result.

17. The method according to claim 16, characterized in that, If the graphic code recognition result is correct, the method further includes: The preset scene set is updated based on the acquired images and the feature extraction results of the acquired images.

18. The method according to claim 17, characterized in that, The scene images in the preset scene set include the corresponding number of correct matches; the method further includes: The scene images in the preset scene set are filtered based on the number of correct matches corresponding to the scene images in the preset scene set.

19. A training device for a feature extraction model, characterized in that, include: The first training module is used to train the first model to be trained based on the first training image set. The first model includes a cascaded first feature extraction module and a second feature extraction module. The first feature extraction module includes a cascaded point convolution module and a depth convolution module. The processing module is used to replace the second feature extraction module in the trained first model with the feature mapping module to obtain the second model. The second training module is used to train the second model using the first training image set, fuse the point convolution module and the depth convolution module of the first feature extraction module in the trained second model, and determine the fused second model as the target model. The second training module is configured to execute: The first feature extraction module in the second model is used to extract features from the images in the first training image set to obtain a feature map containing multiple feature vectors. The feature map is processed by the feature mapping module in the second model to obtain the feature weights corresponding to each feature vector, and the target feature vector is determined from the plurality of feature vectors based on the feature weights. The parameters of the second model are adjusted based on the target feature vector to obtain the trained second model; The point convolution module and the depth convolution module of the first feature extraction module in the trained second model are fused together, and the fused second model is determined as the target model.

20. An information recommendation device, characterized in that, include: The target detection module is used to acquire the collected image and perform target detection on the acquired image based on the object detection model to obtain the graphic code detection result; The feature extraction module is used to perform feature extraction on the acquired image based on a target model if it is determined that the acquired image contains a graphic code according to the graphic code detection result, and to obtain the feature extraction result. The target model is obtained by training a first model based on the feature extraction model training method according to any one of claims 1 to 7. The recommendation module is used to perform scene retrieval based on the feature extraction results, and to recommend graphic codes based on the scene retrieval results.

21. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to perform the method of any one of claims 1 to 18 by executing the executable instructions.

22. A computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 18.

Images