Scene adaptive feature extraction method, electronic device and readable storage medium

Abstract

This application discloses a scene-adaptive feature extraction method, an electronic device, and a readable storage medium. The method includes: acquiring a trained initial text module and an initial image module; obtaining target text and a target image of a target scene; inputting the target text into the initial text module to obtain target text features; constructing a second feature branch parallel to the first feature branch of the initial image module to obtain an adaptive image module; adjusting the second feature branch using the target image and target text features to obtain a trained adaptive image module; obtaining feature correlation between the training loss and the target text features and the target image after passing through the first and second feature branches respectively; inputting the image to be processed collected in the target scene into the adaptive image module to obtain and fuse the features output by the first and second feature branches respectively to obtain the target image features of the image to be processed. This approach can improve the generalization of feature extraction in multiple scenes.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a scene-adaptive feature extraction method, an electronic device, and a readable storage medium. Background Technology

[0002] With the continuous development of smart devices, a large amount of data needs to be processed and analyzed. Among these, the extraction of image features corresponding to images collected in different scenarios is receiving increasing attention. Current technologies typically require training a large number of training samples collected in each scenario to train a model matching each scenario separately. Therefore, when the feature extraction network in a model trained in any scenario is applied to multiple different scenarios, it leads to reduced accuracy in downstream tasks and poor generalization of feature extraction across multiple scenarios. In view of this, improving the generalization of feature extraction across multiple scenarios has become an urgent problem to be solved. Summary of the Invention

[0003] The main technical problem addressed by this application is to provide a scene-adaptive feature extraction method, an electronic device, and a readable storage medium that can improve the generalization of feature extraction in multiple scenarios.

[0004] To address the aforementioned technical problems, this application provides a scene-adaptive feature extraction method, comprising: acquiring a trained image-text matching model; wherein the image-text matching model includes an initial text module and an initial image module; in response to acquiring target text and target image of a target scene, inputting the target text into the initial text module to obtain target text features; constructing a second feature branch parallel to the first feature branch of the initial image module in the initial image module to obtain an adaptive image module; adjusting the parameters of the second feature branch using the target image and the target text features to obtain a trained adaptive image module; wherein the training loss is correlated with the target text features and the features obtained by the target image after passing through the first feature branch and the second feature branch respectively; inputting the image to be processed collected in the target scene into the trained adaptive image module to obtain and fuse the features output by the first feature branch and the second feature branch respectively to obtain the target image features of the image to be processed.

[0005] To address the aforementioned technical problems, a second aspect of this application provides an electronic device comprising: a memory and a processor coupled to each other, wherein the memory stores program data, and the processor invokes the program data to execute the method described in the first aspect.

[0006] To address the aforementioned technical problems, a third aspect of this application provides a computer-readable storage medium storing program data thereon, wherein the program data, when executed by a processor, implements the method described in the first aspect.

[0007] The above scheme obtains a pre-trained image-text matching model, which includes an initial text module and an initial image module. The initial text module extracts text features from text describing the source scene, which are matched with image features extracted from images captured from the source scene by the initial image module. When the target text of the target scene is obtained, the initial text module is used to extract text features from the target text to obtain target text features. The initial image module is then modified to construct a second feature branch parallel to the first feature branch, resulting in an adaptive image module. The adaptive image module is trained using the target image and target text features, and the parameters of the second feature branch are adjusted to obtain the trained adaptive image module. The training loss is correlated with the target text features and the target image features obtained after passing through the first and second feature branches, respectively. The deep semantics corresponding to the features output by the second feature branch of the target text features are matched with the target scene. The matching degree is constrained to enable the second feature branch in the trained adaptive image module to adaptively match the target scene. The feature difference between the output features of the first and second feature branches constrains the shallow semantics included in the output features of the second feature branch, reducing the probability of the second feature branch overfitting the target image and thus improving the generality of the features extracted by the second feature branch. After obtaining the image to be processed from the target scene, the image to be processed is input into the trained adaptive image module to obtain the output features of the first and second feature branches and perform fusion processing to obtain the target image features of the image to be processed. This makes the target image features compatible with the features output by the two feature branches. Using the fused features can improve the accuracy of downstream tasks. Furthermore, by constructing and training the second feature branch, the extracted features can be generalized in different scenes, enabling rapid deployment in different scenes and improving the generalization of feature extraction in different scenes. Attached Figure Description

[0008] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0009] Figure 1 This is a flowchart illustrating one implementation method of the adaptive feature extraction method for the scenario in this application.

[0010] Figure 2 This is a flowchart illustrating another implementation of the adaptive feature extraction method for the scenario described in this application.

[0011] Figure 3 This is a schematic diagram of the structure of one embodiment of the electronic device of this application;

[0012] Figure 4 This is a schematic diagram of one embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0013] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0014] In this paper, the terms "system" and "network" are often used interchangeably. The term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "many" in this paper means two or more.

[0015] The scene-adaptive feature extraction method provided in this application is used to extract features from images to be processed collected in different scenes. The corresponding execution subject is a processing terminal that can call the model and image data.

[0016] Please see Figure 1 , Figure 1 This is a flowchart illustrating one implementation of the adaptive feature extraction method for this application, which includes:

[0017] S101: Obtain the trained image-text matching model, which includes an initial text module and an initial image module.

[0018] Specifically, a pre-trained image-text matching model is obtained, which includes an initial text module and an initial image module. The text features extracted by the initial text module for the text describing the source scene are matched with the image features extracted by the initial image module for the image collected from the source scene.

[0019] In one application, the image-text matching model corresponds to a source scene. The training data includes training text corresponding to the source scene and multiple training images collected from the source scene. The image-text matching model is trained in an unsupervised manner to obtain the trained image-text matching model. During the training process, the loss value is determined based on the matching degree between the training text features extracted from the training text by the initial text module and the training image features extracted from the training images by the initial image module.

[0020] In another application, the image-text matching model corresponds to a source scene. The training data includes multiple training images collected from the source scene and training text corresponding to each training image. The image-text matching model is trained in a semi-supervised manner. Each training text corresponds to labeled reference text features. During the training process, the loss value is determined based on the feature difference between the training text features of the training text and the reference text features extracted by the initial text module, and the matching degree between the training image features of the training image and the reference text features extracted by the initial image module.

[0021] S102: In response to obtaining the target text and target image of the target scene, the target text is input into the initial text module to obtain the target text features. In the initial image module, a second feature branch parallel to the first feature branch of the initial image module is constructed to obtain the adaptive image module.

[0022] Specifically, after obtaining the target text of the target scene, the text features of the target text are extracted using the initial text module to obtain the target text features. The initial image module is then modified to construct a second feature branch parallel to the first feature branch therein, thus obtaining the adaptive image module.

[0023] Understandably, the target scene differs from the source scene corresponding to the image-text matching model. The target text includes text describing the target scene, and the target image includes a small number of reference images collected from the target scene.

[0024] It should be noted that the differences between scenes include at least the differences in scene location, scene lighting, scene weather, and scene background.

[0025] Furthermore, after obtaining the target text and target image of the target scene, the initial text module in the image-text matching model has the ability to extract text features. By inputting the target text into the initial text module, the target text features corresponding to the target text are obtained, where the target text features can describe the target scene.

[0026] Furthermore, the initial image module is modified to construct a second feature branch parallel to the first feature branch in the initial image module, thereby transforming the initial image module into an adaptive image module.

[0027] In one application, a second feature branch with the same network layer structure and parameters as the first feature branch is constructed and placed in a position parallel to the first feature branch to obtain an adaptive image module, which improves the adjustment efficiency when adjusting the second feature branch in the future.

[0028] In another application, a copy of the first feature branch is constructed and some of its parameters are randomly adjusted to obtain a second feature branch with the same network layer structure as the first feature branch. The second feature branch is then placed in a position parallel to the first feature branch to obtain an adaptive image module, so that the feature differences between the output features of the first and second feature branches can be obtained during the initial training.

[0029] S103: Adjust the parameters of the second feature branch using the features of the target image and the target text to obtain the trained adaptive image module. The training loss is related to the target text features and the features of the target image obtained after passing through the first feature branch and the second feature branch, respectively.

[0030] Specifically, the adaptive image module is trained using the features of the target image and the target text, and the parameters of the second feature branch are adjusted to obtain the trained adaptive image module. The training loss is related to the features of the target text and the target image after passing through the first feature branch and the second feature branch, respectively.

[0031] It should be noted that the training loss is related to the difference between the features output by the second feature branch and the target text features. The target text features constrain the matching degree between the deep semantics corresponding to the features output by the second feature branch and the target scene, so that the second feature branch in the trained adaptive image module can adaptively match the target scene.

[0032] Furthermore, the adaptive image module differs little from the initial image module, requiring only a small number of network layers to train. Therefore, the adaptive image module can be trained using only a small number of target images, reducing its training difficulty. The training loss is also related to the feature differences between the output features of the first and second feature branches. These differences constrain the shallow semantics included in the features output by the second feature branch, reducing the probability of overfitting the target image and thus improving the generality of the features extracted by the second feature branch. The shallow semantics are at least related to the image's texture features.

[0033] In one application, the target image is input into the adaptive image module to obtain the features output by the first feature branch and the second feature branch respectively. The training loss is determined based on the feature difference between the features output by the second feature branch and the target text features, as well as the features output by the first feature branch and the second feature branch respectively. The parameters of the second feature branch are adjusted using the training loss until the convergence condition is met, and the trained adaptive image module is obtained.

[0034] In one application scenario, the convergence condition includes training loss being less than a loss threshold.

[0035] In another application scenario, the convergence conditions include training loss being less than a loss threshold and the number of training iterations being greater than a number of iterations threshold.

[0036] S104: Input the image to be processed collected in the target scene into the trained adaptive image module, obtain the features output by the first feature branch and the second feature branch respectively, and fuse them to obtain the target image features of the image to be processed.

[0037] Specifically, after obtaining the image to be processed from the target scene, the image to be processed is input into the trained adaptive image module to obtain the features output by the first feature branch and the second feature branch respectively, and then perform fusion processing to obtain the target image features of the image to be processed. This makes the target image features compatible with the features output by the two feature branches, thereby improving the generalization of the target image features.

[0038] It should be noted that the first feature branch in the adaptive image module originates from the first feature branch in the initial image module, and its network layer structure and parameters are not adjusted. Therefore, the parameters of the first feature branch have a higher matching degree with the source scene of the image-text matching model. The second feature branch, after training, theoretically has a higher matching degree with the target scene. However, the second feature branch has a probability of overfitting to the target image during the training phase. Therefore, by fusing the features output by the first and second feature branches, the difference between the target image features and the features extracted by the first feature branch can be reduced, thus reducing the probability of weakened feature generalization due to overfitting. At the same time, the target image features can also match the target scene.

[0039] In one application method, the first weight corresponding to the first feature branch and the second weight corresponding to the second feature branch are obtained, wherein the first weight is less than the second weight. The first image features and the second image features are weighted and summed to obtain the target image features of the image to be processed.

[0040] In another application, the second weight corresponding to the second feature branch is determined based on the matching degree between the features output by the second feature branch and the target text features. Based on the total weight and the second weight, the first weight corresponding to the first feature branch is determined. The first image features and the second image features are weighted and summed to obtain the target image features of the image to be processed.

[0041] Optionally, the acquired target image features can be applied to downstream tasks such as classification, segmentation, and object detection. Using the fused features can improve the accuracy of downstream tasks. Furthermore, by constructing and training a second feature branch, the extracted features can be made universal in different scenarios, enabling rapid deployment in different scenarios and improving the generalization of feature extraction in different scenarios.

[0042] The above scheme obtains a pre-trained image-text matching model, which includes an initial text module and an initial image module. The initial text module extracts text features from text describing the source scene, which are matched with image features extracted from images captured from the source scene by the initial image module. When the target text of the target scene is obtained, the initial text module is used to extract text features from the target text to obtain target text features. The initial image module is then modified to construct a second feature branch parallel to the first feature branch, resulting in an adaptive image module. The adaptive image module is trained using the target image and target text features, and the parameters of the second feature branch are adjusted to obtain the trained adaptive image module. The training loss is correlated with the target text features and the target image features obtained after passing through the first and second feature branches, respectively. The deep semantics corresponding to the features output by the second feature branch of the target text features are matched with the target scene. The matching degree is constrained to enable the second feature branch in the trained adaptive image module to adaptively match the target scene. The feature difference between the output features of the first and second feature branches constrains the shallow semantics included in the output features of the second feature branch, reducing the probability of the second feature branch overfitting the target image and thus improving the generality of the features extracted by the second feature branch. After obtaining the image to be processed from the target scene, the image to be processed is input into the trained adaptive image module to obtain the output features of the first and second feature branches and perform fusion processing to obtain the target image features of the image to be processed. This makes the target image features compatible with the features output by the two feature branches. Using the fused features can improve the accuracy of downstream tasks. Furthermore, by constructing and training the second feature branch, the extracted features can be generalized in different scenes, enabling rapid deployment in different scenes and improving the generalization of feature extraction in different scenes.

[0043] Please see Figure 2 , Figure 2This is a flowchart illustrating another implementation of the adaptive feature extraction method for this application, which includes:

[0044] S201: Obtain the trained image-text matching model, which includes an initial text module and an initial image module.

[0045] Specifically, the image-text matching model corresponds to the source scene and is pre-trained. The text features extracted by the initial text module after training for the text describing the source scene are matched with the image features extracted by the initial image module after training for the image collected from the source scene.

[0046] Optionally, the training process of the image-text matching model includes: acquiring multiple training images of the source scene and training text corresponding to each training image; inputting all training text and all training images into the image-text matching model to obtain the training text features of each training text output by the initial text module and the training image features of each training image output by the initial image module; determining the text matching loss based on the accuracy of matching training image features with each training text feature; determining the image matching loss based on the accuracy of matching training text features with each training image feature; and adjusting the parameters of the initial text module and the initial image module based on the text matching loss and the image matching loss until the convergence condition is met to obtain the trained image-text matching model.

[0047] Specifically, training data corresponding to the source scene is obtained. The training data includes a large number of training images collected from the source scene and training text corresponding to each training image. The training text is related to the location and background of the source scene, as well as the lighting and weather of the training images.

[0048] Understandably, all training texts and all training images are input into the image-text matching model so that the initial text module extracts features from each training text, obtaining the training text features of each training text output by the initial text module, and the initial image module extracts features from each training image, obtaining the training image features of each training image output by the initial image module.

[0049] Furthermore, the training image features matched by each training text feature are obtained, the accuracy of matching each training text feature with the training image feature is determined, the text matching loss is determined, the training text features matched by each training image feature are obtained, the accuracy of matching each training image feature with the training text feature is determined, the image matching loss is determined, and based on the text matching loss and the image matching loss, the model training loss is determined, thereby improving the accuracy of the model training loss. The parameters of the initial text module and the initial image module are adjusted until the convergence condition is met, and the trained image-text matching model is obtained.

[0050] In one application scenario, all training texts and training images are input into the image-text matching model to obtain the training text features F corresponding to each training text output by the initial text module. text And the training image features F corresponding to each training image output by the initial image module. image The training text features F text and training image features F image After normalization, the normalized text features F are obtained. tnorm and normalized image features F inorm Normalized text features F tnorm Transpose representation of normalized image features Matrix multiplication yields the probability representation of each training text matching the training image; conversely, normalizing the image features F... inorm Transpose representation of normalized text features Perform matrix multiplication to obtain the probability representation of each training image matching the training text, and calculate the cross-entropy loss to obtain the text matching loss. text Image matching loss image The two losses are combined and backpropagated to adjust the parameters of the initial text module and the initial image module until the convergence condition is met, thus obtaining the trained image-text matching model.

[0051] S202: In response to obtaining the target text and target image of the target scene, the target text is input into the initial text module to obtain the target text features. In the initial image module, a second feature branch parallel to the first feature branch of the initial image module is constructed to obtain the adaptive image module.

[0052] Specifically, after obtaining the target text and target image of the target scene, the target text is input into the initial text module to obtain the target text features corresponding to the target text output by the initial text module. In the initial image module, a second feature branch parallel to the first feature branch of the initial image module is constructed to obtain the adaptive image module.

[0053] In one application scenario, in the initial image module, a second feature branch is constructed parallel to the first feature branch of the initial image module to obtain an adaptive image module. This includes: constructing a copy of the first feature branch based on the network layer structure and parameters of the first feature branch in the initial image module to obtain the second feature branch; and setting the second feature branch in the initial image module in a position parallel to the first feature branch to obtain the adaptive image module.

[0054] Specifically, the first feature branch in the initial image module is extracted, and a copy of the first feature branch is constructed according to the network layer structure and parameters of the first feature branch. The copy of the first feature branch is used as the second feature branch, so that the initially constructed second feature branch is the same as the first feature branch, reducing the difficulty of subsequent adjustments to the second feature branch. A second feature branch is added to the initial image module and set in a position parallel to the first feature branch to obtain an adaptive image module. Thus, when the input image is obtained, the input image can pass through the first feature branch and the second feature branch respectively to obtain the features output by the first feature branch and the second feature branch respectively.

[0055] In a specific application scenario, the first feature branch includes multiple cascaded feedforward network layers. A copy of the first feature branch is generated to obtain a second feature branch with the same cascading method and parameters as the feedforward network layers of the first feature branch. The second feature branch is then placed in a position parallel to the first feature branch to obtain an adaptive image module.

[0056] S203: Input the target image into the adaptive image module to obtain the first training feature output by the first feature branch and the second training feature output by the second feature branch.

[0057] Specifically, the target image is input into the adaptive image module so that the target image passes through the first feature branch and the second feature branch in the adaptive image module to obtain the first training feature output by the first feature branch and the second training feature output by the second feature branch.

[0058] S204: The first loss is obtained based on the first training feature and the second training feature, and the second loss is obtained based on the second training feature and the target text feature.

[0059] Specifically, a first loss is obtained based on the feature difference between the first and second training features, and a second loss is obtained based on the feature difference between the second training features and the target text features. The first loss characterizes the difference between the second training features and the first training features in the dimension of image texture, corresponding to shallow semantics; the second loss characterizes the difference between the second training features and the target text features in the dimension of image semantics, corresponding to deep semantics.

[0060] In one application scenario, the first feature branch and the second feature branch include the same number of cascaded feedforward network layers. In the first feature branch, the feedforward network layers at the beginning of the layer output the first sub-feature in sequence until the last feedforward network layer outputs the first training feature. In the second feature branch, the feedforward network layers at the beginning of the layer output the second sub-feature in sequence until the last feedforward network layer outputs the second training feature.

[0061] Further, the first loss is obtained based on the first training feature and the second training feature, including: obtaining the first loss based on the feature difference between the first sub-feature and the second sub-feature that are the same at all levels, and the feature difference between the first training feature and the second training feature.

[0062] Specifically, the first feature branch and the second feature branch each include an equal number of cascaded feedforward network layers. When the target image input passes through the first feature branch and the second feature branch, in the first feature branch, the first sub-feature output by the first feedforward network layer at the beginning of the layer is used as the input of the next feedforward network layer, until the last feedforward network layer outputs the first training feature. Similarly, in the second feature branch, the second sub-feature output by the first feedforward network layer at the beginning of the layer is used as the input of the next feedforward network layer, until the last feedforward network layer outputs the second training feature.

[0063] Furthermore, by utilizing the feature difference between the first and second sub-features that are identical across all levels, as well as the feature difference between the first and second training features, a first loss is obtained. Based on the first loss, backpropagation is performed to adjust the parameters of the second feature branch, thereby improving the training effect of the adaptive image module.

[0064] S205: Based on the first loss and the second loss, determine the training loss, and adjust the parameters of the second feature branch based on the training loss until the convergence condition is met, thus obtaining the trained adaptive image module.

[0065] Specifically, based on the first loss and the second loss, the training loss of the adaptive image module is determined, and the parameters of the second feature branch in the adaptive image module are adjusted based on the training loss until the convergence condition is met, thus obtaining the trained adaptive image module.

[0066] In one application scenario, a training loss is determined based on a first loss and a second loss, and the parameters of the second feature branch are adjusted based on the training loss until a convergence condition is met to obtain a trained adaptive image module. This includes: weighted summation of the first and second losses to obtain the training loss; adjusting the parameters of the second feature branch based on the training loss to obtain an updated second feature branch; returning to the step of inputting the target image into the adaptive image module to obtain the first training feature output by the first feature branch and the second training feature output by the second feature branch; and fixing the parameters of the second feature branch in response to the training loss being less than a loss threshold to obtain the trained adaptive image module.

[0067] Specifically, the first loss and the second loss are weighted and summed to obtain the training loss, which can be customized based on the application scenario. When the first loss is obtained based on the feature difference between the first and second sub-features of the same level across all layers, and the feature difference between the first and second training features, the training loss is expressed by the following formula:

[0068]

[0069] Where Loss represents the training loss, Loss text Indicates the second loss. This represents the feature difference between the features output by each feedforward network layer, where n represents the number of feedforward network layers.

[0070] Furthermore, backpropagation is performed based on the training loss to adjust the parameters of the second feature branch, resulting in an updated second feature branch. The process then returns to the step of inputting the target image into the adaptive image module to obtain the first training feature output by the first feature branch and the second training feature output by the second feature branch, until the training loss is less than the loss threshold. The parameters of the second feature branch are then fixed to obtain the trained adaptive image module, making the second feature branch in the adaptive image module more accurate.

[0071] S206: Input the image to be processed collected in the target scene into the trained adaptive image module, obtain the features output by the first feature branch and the second feature branch respectively, and fuse them to obtain the target image features of the image to be processed.

[0072] Specifically, the parameters of the first feature branch remain unchanged, the first feature branch matches the source scene, the second feature branch matches the target scene, and the source scene and the target scene correspond to the same feature extraction task.

[0073] In one application scenario, the image to be processed, collected from the target scene, is input into the trained adaptive image module to obtain the features output by the first feature branch and the second feature branch respectively, and then fused to obtain the target image features of the image to be processed. This includes: inputting the image to be processed, collected from the target scene, into the trained adaptive image module to obtain the first image features output by the first feature branch and the second image features output by the second feature branch; and performing a weighted summation of the first image features and the second image features to obtain the target image features of the image to be processed.

[0074] Specifically, the image to be processed, collected from the target scene, is input into the trained adaptive image module, so that the first feature branch and the second feature branch in the adaptive image module extract features from the image to be processed, respectively, to obtain the first image features output by the first feature branch and the second image features output by the second feature branch.

[0075] Furthermore, the first image features and the second image features are weighted and summed so that the first image features and the second image features can be fused according to the set weights to obtain the target image features of the image to be processed, making the target image features more accurate.

[0076] It should be noted that when the image to be processed collected in the target scene is input into the adaptive image module, the first feature branch outputs the first image feature and the second feature branch outputs the second image feature. Both the first image feature and the second image feature can characterize the feature information of the image to be processed. After fusing the first image feature and the second image feature, the target image feature corresponding to the image to be processed is obtained. Thus, the second image feature is corrected by using the first image feature through fusion, reducing the probability of the generalization of the target image feature weakening due to the second feature branch overfitting the target image.

[0077] In a specific application scenario, the target image features of the image to be processed are obtained by weighted summation of the first image features and the second image features. This includes: obtaining the similarity between the second image features and the target text features; determining the assigned weights of the first image features and the second image features based on the similarity and the preset total weight; wherein the assigned weight of the second image features is positively correlated with the similarity; and using the assigned weights of the first image features and the second image features, the target image features of the image to be processed are obtained by weighted summation of the first image features and the second image features.

[0078] Specifically, the similarity between the second image features and the target text features is obtained. The assigned weight of the second image features is positively correlated with the similarity. Therefore, the higher the similarity, the greater the assigned weight of the second image features and the higher its proportion in the target image features.

[0079] Optionally, assuming the similarity is between 0 and 1, the total weight is 1. After obtaining the assigned weights for the second image features, the assigned weights for the first image features can be obtained. Therefore, the first image features are negatively correlated with the similarity. Using the assigned weights for the first and second image features respectively, the first and second image features are weighted and summed to obtain the target image features of the image to be processed, making the target image features more accurate. The above process is expressed by the following formula:

[0080] F=(1-α)*F1+a*F2 (2)

[0081] Where F represents the target image feature, F1 represents the first image feature, F2 represents the second image feature, and α represents the similarity.

[0082] Unlike the previous embodiments, the first feature branch and the second feature branch include multiple cascaded forward feedback network layers of the same number. A first loss is obtained by using the feature difference between the first and second sub-features of the same level and the feature difference between the first and second training features. The parameters of the second feature branch are adjusted by backpropagation based on the first loss to improve the training effect of the adaptive image module. The first image features and the second image features are weighted and summed so that the first image features and the second image features can be fused according to the set weights. The weight assigned to the second image features is positively correlated with the similarity. Therefore, when the similarity is higher, the weight assigned to the second image features is larger and the proportion in the target image features is higher. Both the first image features and the second image features can represent the feature information of the image to be processed. After fusing the first image features and the second image features, the target image features corresponding to the image to be processed are obtained. Thus, the second image features are corrected by using the first image features through fusion, reducing the probability of the target image features weakening due to the second feature branch overfitting the target image.

[0083] Please see Figure 3 , Figure 3 This is a schematic diagram of the structure of an embodiment of the electronic device of this application. The electronic device 30 includes a memory 301 and a processor 302 coupled to each other. The memory 301 stores program data (not shown in the figure), and the processor 302 calls the program data to implement the method in any of the above embodiments. For the description of the relevant content, please refer to the detailed description of the above method embodiments, which will not be repeated here.

[0084] Please see Figure 4 , Figure 4 This is a schematic diagram of a computer-readable storage medium according to an embodiment of the present application. The computer-readable storage medium 40 stores program data 400. When the program data 400 is executed by a processor, it implements the method in any of the above embodiments. For a detailed description of the relevant content, please refer to the detailed description of the above method embodiments, which will not be repeated here.

[0085] It should be noted that the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0086] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0087] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0088] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A scene-adaptive feature extraction method, characterized in that, include: Obtain the trained image-text matching model; wherein the image-text matching model includes an initial text module and an initial image module; In response to obtaining the target text and target image of the target scene, the target text is input into the initial text module to obtain the target text features. In the initial image module, a second feature branch parallel to the first feature branch of the initial image module is constructed to obtain the adaptive image module. The parameters of the second feature branch are adjusted using the target image and the target text features to obtain a trained adaptive image module. The training loss is related to the target text features and the features obtained after the target image passes through the first and second feature branches, respectively. The adaptive image module is trained based on the following steps: inputting the target image into the adaptive image module to obtain a first training feature output by the first feature branch and a second training feature output by the second feature branch; obtaining a first loss based on the first and second training features, and obtaining a second loss based on the second training features and the target text features; determining the training loss based on the first and second losses, and adjusting the parameters of the second feature branch based on the training loss until a convergence condition is met, thus obtaining the trained adaptive image module. The image to be processed, collected from the target scene, is input into the trained adaptive image module to obtain the features output by the first feature branch and the second feature branch respectively, and then fused to obtain the target image features of the image to be processed.

2. The scene adaptive feature extraction method according to claim 1, characterized in that, The first feature branch and the second feature branch each include a number of sequentially cascaded feedforward network layers. In the first feature branch, the feedforward network layers at the beginning of the layer output the first sub-feature sequentially until the last feedforward network layer outputs the first training feature. In the second feature branch, the feedforward network layers at the beginning of the layer output the second sub-feature sequentially until the last feedforward network layer outputs the second training feature. The first loss is obtained based on the first training feature and the second training feature, including: The first loss is obtained based on the feature difference between the first sub-feature and the second sub-feature that are the same at all levels, and the feature difference between the first training feature and the second training feature.

3. The scene adaptive feature extraction method according to claim 1, characterized in that, The step of determining the training loss based on the first loss and the second loss, and adjusting the parameters of the second feature branch based on the training loss until the convergence condition is met, to obtain the trained adaptive image module, includes: The training loss is obtained by weighted summation of the first loss and the second loss. The parameters of the second feature branch are adjusted based on the training loss to obtain the updated second feature branch, and the process returns to the step of inputting the target image into the adaptive image module to obtain the first training feature output by the first feature branch and the second training feature output by the second feature branch; In response to the training loss being less than the loss threshold, the parameters of the second feature branch are fixed to obtain the trained adaptive image module.

4. The scene adaptive feature extraction method according to any one of claims 1-3, characterized in that, The step of constructing a second feature branch parallel to the first feature branch of the initial image module in the initial image module to obtain an adaptive image module includes: Based on the network layer structure and parameters of the first feature branch in the initial image module, a copy of the first feature branch is constructed to obtain the second feature branch; In the initial image module, the second feature branch is positioned parallel to the first feature branch to obtain the adaptive image module.

5. The scene adaptive feature extraction method according to claim 1, characterized in that, The step of inputting the image to be processed collected in the target scene into the trained adaptive image module, obtaining the features output by the first feature branch and the second feature branch respectively, and fusing them to obtain the target image features of the image to be processed includes: The image to be processed, collected from the target scene, is input into the trained adaptive image module to obtain the first image feature output by the first feature branch and the second image feature output by the second feature branch. The first image feature and the second image feature are weighted and summed to obtain the target image feature of the image to be processed.

6. The scene adaptive feature extraction method according to claim 5, characterized in that, The step of weighted summing of the first image features and the second image features to obtain the target image features of the image to be processed includes: The similarity between the second image feature and the target text feature is obtained. Based on the similarity and a preset total weight, the allocation weights corresponding to the first image feature and the second image feature are determined respectively. The allocation weight corresponding to the second image feature is positively correlated with the similarity. By using the assigned weights of the first image feature and the second image feature respectively, a weighted sum is performed on the first image feature and the second image feature to obtain the target image feature of the image to be processed.

7. The scene adaptive feature extraction method according to claim 1, characterized in that, The training process of the image-text matching model includes: Multiple training images of the source scene and training text corresponding to each training image are obtained. All training texts and all training images are input into the image-text matching model to obtain the training text features of each training text output by the initial text module and the training image features of each training image output by the initial image module. Based on the accuracy of matching each training text feature with the training image feature, determine the text matching loss; based on the accuracy of matching each training image feature with the training text feature, determine the image matching loss. Based on the text matching loss and the image matching loss, the parameters of the initial text module and the initial image module are adjusted until the convergence condition is met, thus obtaining the trained image-text matching model.

8. An electronic device, characterized in that, include: A memory and a processor are coupled to each other, wherein the memory stores program data, and the processor invokes the program data to perform the method as described in any one of claims 1-7.

9. A computer-readable storage medium storing program data thereon, characterized in that, When the program data is executed by the processor, the method as described in any one of claims 1-7 is implemented.

Images