Visual feature extraction method and device, equipment, storage medium and program product

Abstract

The invention discloses a visual feature extraction method and device, equipment, a storage medium and a program product, and relates to the technical field of multi-modal large models, and the method comprises the steps: obtaining target media data; performing visual coding on the target media data by using a visual encoder in the multi-modal processing model to obtain a target visual feature; wherein the visual encoder is configured to be a visual feature layer, a connection layer and a multi-modal feature layer, and the visual feature layer, the connection layer and the multi-modal feature layer are obtained through progressive training based on a preset sequence. According to the technical scheme of the invention, the visual features extracted by the visual encoder can be gradually aligned with the feature space of the multi-modal processing model, and the visual understanding capability of the multi-modal processing model is improved.

Description

Technical Field

[0001] The present disclosure relates to the technical field of multimodal large models, and in particular to a method, apparatus, device, storage medium, and program product for extracting visual features. Background Art

[0002] As a key component for acquiring visual information in a large multimodal language model, the performance of the visual encoder directly impacts the model's multimodal understanding capabilities. The pre-training methods (contrastive learning or self-supervision) used by current visual encoders (ViTs) are misaligned with the optimization goals of large multimodal language models. Furthermore, their training data types are relatively limited, resulting in a significant gap between the extracted visual features and the feature space of large multimodal language models, leading to poor visual understanding capabilities of large multimodal language models. Summary of the Invention

[0003] In view of this, the present disclosure provides a method, apparatus, device, storage medium and program product for extracting visual features to solve the problem that the visual features extracted by the visual encoder are difficult to align with the multimodal large language model.

[0004] In a first aspect, the present disclosure provides a method for extracting visual features, comprising: obtaining target media data; visually encoding the target media data using a visual encoder in a multimodal processing model to obtain target visual features; wherein the visual encoder is configured as a visual feature layer, a connection layer, and a multimodal feature layer, and the visual feature layer, the connection layer, and the multimodal feature layer are obtained by progressive training based on a preset order.

[0005] In a second aspect, the present disclosure provides a visual feature extraction device, comprising: an acquisition module for acquiring target media data; a feature extraction module for inputting the target media data into a visual encoder of a multimodal processing model, and using the visual encoder to visually encode the target media data to obtain target visual features; wherein the visual encoder is configured as a visual feature layer, a connection layer and a multimodal feature layer, and the visual feature layer, the connection layer and the multimodal feature layer are obtained by progressive training based on a preset order.

[0006] In a third aspect, the present disclosure provides an electronic device comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the visual feature extraction method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0007] In a fourth aspect, the present disclosure provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the method for extracting visual features of the first aspect or any corresponding embodiment thereof.

[0008] In a fifth aspect, the present disclosure provides a computer program product, comprising computer instructions for causing a computer to execute the method for extracting visual features of the first aspect or any corresponding embodiment thereof.

[0009] The visual feature extraction method, apparatus, device, storage medium, and program product provided by the embodiments of the present disclosure progressively train a visual encoder configured as a visual feature layer, a connection layer, and a multimodal feature layer in a preset order, so that the visual features extracted by the visual encoder can gradually align with the feature space of the multimodal processing model through progressive training. Subsequently, the target media data is visually encoded according to the trained visual encoder, and the obtained target visual features can be more quickly and better aligned with the feature space of the multimodal processing model, thereby improving the visual understanding ability of the multimodal processing model. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the specific embodiments of the present disclosure or the technical solutions in the related technologies, the following briefly introduces the drawings required for use in the specific embodiments or related technical descriptions. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0011] Figure 1 is a schematic diagram of an application scenario according to an embodiment of the present disclosure;

[0012] Figure 2 is a flowchart of a method for extracting visual features according to an embodiment of the present disclosure;

[0013] Figure 3 is a schematic diagram of a visual encoder according to an embodiment of the present disclosure;

[0014] Figure 4 1 is a schematic diagram of a training process of a visual encoder according to an embodiment of the present disclosure;

[0015] Figure 5 is a schematic diagram of sample data according to the present disclosure;

[0016] Figure 6 is a training diagram of the first training stage of the visual encoder according to the present disclosure;

[0017] Figure 7 is a training diagram of the second training stage of the visual encoder according to the present disclosure;

[0018] Figure 8 is a training diagram of the third training stage of the visual encoder according to the present disclosure;

[0019] Figure 9 Schematic diagram of the lower triangle constraint during the training process of the visual encoder according to the present disclosure;

[0020] Figure 10 This is a comparison diagram of the visual encoder trained according to the present disclosure and other visual encoders;

[0021] Figure 11 is a structural block diagram of a visual feature extraction device according to an embodiment of the present disclosure;

[0022] Figure 12 Schematic diagram of the hardware structure of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0023] To make the purpose, technical solutions, and advantages of the embodiments of the present disclosure more clear, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative efforts shall fall within the scope of protection of the present disclosure.

[0024] It is understandable that before using the technical solutions disclosed in the various embodiments of this disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved in this disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0025] For example, in response to a user's active request, a prompt message is sent to the user to clearly inform the user that the operation requested will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the electronic device, application, server, storage medium, or other software or hardware that performs the operations of the disclosed technical solution based on the prompt message.

[0026] As an optional but non-limiting implementation, in response to receiving a user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. Furthermore, the pop-up window may also contain a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0027] It is understandable that the above notification and user authorization process are merely illustrative and do not limit the implementation of the present disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0028] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) must comply with the requirements of relevant laws, regulations and relevant provisions.

[0029] With the development of multimodal large language models (MLLMs), in the visual processing module, visual transformers (ViTs) are used as the mainstream encoder. They are combined with large language models (LLMs) through connection modules such as Q-former or MLP, giving MLLMs the dual capabilities of text reasoning and visual perception.

[0030] Currently, the visual encoders used in most multimodal processing models based on MLLM are primarily based on two pre-training strategies: contrastive learning and autoregressive modeling. The former relies on large-scale image and text data to extract visual features, while the latter provides more refined supervision signals through a generative loss function.

[0031] However, neither of these training methods can guarantee that the visual features extracted by ViTs are fully aligned with the feature space of the large language model. Furthermore, the ViTs training data used in related technologies is often limited to captions, lacking multimodal information that embodies complex world knowledge, resulting in insufficient generalization of the extracted features.

[0032] Based on this, the technical solution disclosed in the present invention adopts a three-stage feature refinement strategy, namely "from coarse-grained modal alignment, to fine-grained modal alignment, and finally injecting rich world knowledge", realizing cross-modal alignment from coarse to fine, and effectively injecting rich world knowledge, so that the visual features extracted by ViT can be aligned with the feature space of the multimodal processing model faster and better to improve the overall visual understanding ability performance. Experimental verification has determined that the visual encoder disclosed in the present invention has achieved better visual understanding performance than open source visual encoders of the same level after being connected to different multimodal processing models.

[0033] As an optional application scenario of the embodiment of the present disclosure, Figure 1 As shown, the optional application scenario includes a visual encoder 101, a multimodal processing model 102 and an electronic device 103, wherein the visual encoder 101 is connected to the multimodal processing model 102, the multimodal processing model 102 is deployed in a generative application 104 to provide visual content generation, and the generative application 104 is deployed in the electronic device 103.

[0034] The electronic device 103 may be a device with computing capabilities, for example, the electronic device 103 may be provided with a processor and memory, etc., or may be equipped with a dedicated accelerator (such as a graphics processing unit (GPU)). In addition, the electronic device 103 may store and maintain data.

[0035] Examples of electronic devices 103 may include supercomputers, personal computers, laptop computers, in-vehicle computing devices, mobile devices (such as smartphones, tablet computers, etc.), or a combination of any one or more of the above devices. It should be understood that the electronic devices described herein are merely exemplary and non-limiting, and for example, other different types of electronic devices may also be used.

[0036] According to an embodiment of the present disclosure, an embodiment of a method for extracting visual features is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0037] In this embodiment, a method for extracting visual features is provided, which can be used in electronic devices such as computers, tablet computers, etc. Figure 2 is a flow chart of a method for extracting visual features according to an embodiment of the present disclosure, such as Figure 2 As shown, the process includes the following steps:

[0038] Step S201: Acquire target media data.

[0039] The target media data is media data that needs to be encoded, such as image data, video data, etc. Specifically, a generative application is deployed in the electronic device, and the generative application provides an interactive page. The target media data can be input through the interactive page, and accordingly, the generative application can obtain the media data.

[0040] The target media data may be selected and uploaded to the interactive page from the local electronic device, or may be captured and uploaded to the interactive page by calling a camera function. The method for uploading the target media data is not specifically limited here.

[0041] Step S202 : Visually encode the target media data using a visual encoder in the multimodal processing model to obtain target visual features.

[0042] Among them, the visual encoder is configured as a visual feature layer, a connection layer and a multimodal feature layer, and the visual feature layer, the connection layer and the multimodal feature layer are obtained by progressive training based on a preset order.

[0043] A multimodal processing model is a model that can process and integrate multiple different types of data (such as text, images, audio, video, etc.). The multimodal processing model can learn the representations between different modalities, so that the model can understand and process the relationships between different modalities, and better understand context and complex scenarios by integrating information from different modalities. Specifically, the multimodal processing model can be trained based on a large language model architecture, or it can be trained based on a machine learning model architecture. As long as it can realize the processing of multimodal data, the model architecture of the multimodal processing model is not limited here.

[0044] The multimodal processing model includes a visual encoder, which extracts features from image or video content. Specifically, the multimodal processing model is deployed within a generative application to provide generated content for the generative application. When content generation is required for target media data, the target media data is input into the multimodal processing model's visual encoder for visual encoding, extracting visual features from the target media data. This visual features are then used to generate relevant content.

[0045] As a network model for extracting image or video content features, the visual encoder is configured with a visual feature layer, a connection layer, and a multimodal feature layer, such as Figure 3 As shown in the figure, the visual feature layer is used to extract image features or video features; the connection layer is used to map the features of the visual feature layer to the multimodal feature layer; and the multimodal feature layer is used to fuse visual features with other modal knowledge to better understand the content features carried by the image or video.

[0046] Specifically, the visual feature layer, the connection layer, and the multimodal feature layer are not trained jointly at the same time. Instead, they are trained in a progressive manner according to a preset order (pre-set training order). The learning hyperparameters of the visual feature layer, the connection layer, and the multimodal feature layer are trained in stages in the preset order to achieve cross-modal alignment from coarse to fine, so as to achieve visual feature refinement in each training stage of the visual feature layer, the connection layer, and the multimodal feature layer.

[0047] The visual feature extraction method provided in this embodiment progressively trains a visual encoder configured as a visual feature layer, a connection layer, and a multimodal feature layer in a predetermined order. This progressive training method enables the visual features extracted by the visual encoder to gradually align with the feature space of the multimodal processing model. Subsequently, the target media data is visually encoded using the trained visual encoder. The resulting target visual features can be more quickly and effectively aligned with the feature space of the multimodal processing model, thereby enhancing the visual understanding capabilities of the multimodal processing model.

[0048] In this embodiment, a method for progressively training a visual encoder based on a preset sequence is provided. The visual encoder is applied to the multimodal processing model for visual encoding. Figure 4 is a flow chart of a method for progressively training a visual encoder based on a preset sequence according to an embodiment of the present disclosure, such as Figure 4 As shown, the process includes the following steps:

[0049] Step S301 : Using the first sample data to train the connection layer, so that the pre-trained connection layer aligns the feature dimensions of the visual feature layer and the multimodal feature layer.

[0050] The first sample data is pre-set sample data for training connection layer parameters. Specifically, the first sample data includes graphic and text sample data for describing an image and image and text recognition data obtained by recognizing the image.

[0051] During the training process of the connection layer, the first sample data is used to train the learning hyperparameters of the connection layer so that the connection layer can perform coarse-grained alignment of the feature dimensions of the visual information extracted by the visual feature layer and the text semantics recognized by the multimodal feature layer based on large-scale perception data, which serves as the basis for the second stage of training the visual feature layer to perform fine-grained feature alignment.

[0052] Step S302 : jointly train the visual feature layer using the second sample data and the pre-trained connection layer, so that the visual feature layer can extract multimodal visual features.

[0053] The second sample data is pre-set sample data for jointly training the learning hyperparameters of the connection layer and the visual feature layer. Specifically, the second sample data has a larger data volume than the first sample data, and compared with the first sample data, it additionally adds video sample data to enable the visual feature layer to perform multi-task pre-training.

[0054] After completing the connection layer pre-training in the first stage, the connection layer already has the feature alignment capability. At this time, combined with the second sample data, the learning hyperparameters of the connection layer and the visual feature layer are jointly trained, and the second sample data is used to enhance the visual feature capture capability of the visual feature layer, so that the visual feature layer can perceive and capture more visual data, adapt to the visual feature extraction of multimodal visual data such as image data and video data, and facilitate the connection with the multimodal knowledge injection for the multimodal feature layer in the third stage.

[0055] Step S303 : jointly train the multimodal feature layer using the third sample data, the pre-trained connection layer, and the pre-trained visual feature layer, so as to inject multimodal knowledge into the multimodal feature layer.

[0056] The third sample data is pre-set sample data used to jointly train the learning hyperparameters of the connection layer, visual feature layer, and multimodal feature layer. Specifically, the third sample data is larger in volume than the second sample data and includes multimodal data types, such as graphic and text sample data, image and text recognition data, mathematical knowledge, text data, and question-and-answer data, so that the multimodal feature layer can understand multimodal knowledge.

[0057] After completing the second phase of joint training of the visual feature layer and the connection layer, the connection layer already possesses excellent feature alignment capabilities, and the visual feature layer is already capable of accurately capturing visual elements. At this point, combined with the third sample data containing multimodal data types, the learning hyperparameters of the connection layer, visual feature layer, and multimodal feature layer are jointly trained. By introducing a more diverse multimodal instruction dataset, the multimodal feature layer is infused with more comprehensive world knowledge (i.e., multimodal knowledge), enabling the trained multimodal feature layer to extract effective visual features from diverse visual information.

[0058] The method provided in this embodiment for progressively training a visual encoder based on a preset order performs training of a connection layer, a visual feature layer, and a multimodal feature layer in three stages, thereby achieving progressive training of coarse-grained alignment of feature dimensions, fine-grained alignment of features, and multimodal knowledge injection. This enables cross-modal alignment from coarse to fine, thereby avoiding the impact of feature misalignment between the visual encoder and the multimodal processing model on visual comprehension capabilities.

[0059] In some optional implementations, the first sample data includes first image sample data, first graphic and text sample data, and first image and text recognition data, and the first graphic and text sample data and the first image and text recognition data are matched according to a first preset ratio.

[0060] As described above, the first sample data is used to train the learning hyperparameters of the connection layer, and the first sample data includes first image sample data, first graphic and text sample data, and first image and text recognition data. Figure 6 As shown, the first image sample data is input into the visual feature layer, which then uses the visual feature layer to extract visual information from the first image sample data. The first image and text sample data and the first image text recognition data are then input into the multimodal feature layer, which combines the image and text sample data and the instructions corresponding to the image and text data to recognize text semantics. The connection layer can then combine visual information and text semantics for feature mapping, achieving coarse-grained alignment of visual information and text semantics in the feature dimension.

[0061] The first image and text sample data and the first image and text recognition data are matched according to a first preset ratio (pre-set data ratio), and the first image and text sample data accounts for a larger proportion than the first image and text recognition data. Figure 5 As shown, different sample data are extracted from different data sources. The total amount of the first graphic sample data and the first image text recognition data is 8M, of which the first graphic sample data (graphic caption data) accounts for 4.9M and the first image text recognition data (i.e., image OCR data) accounts for 3.1M.

[0062] The second sample data includes second image sample data, second video sample data, second graphic and text sample data, second video description sample data and second image text recognition data. The second graphic and text sample data, second video description sample data and second image text recognition data are matched according to a second preset ratio, and the data volume of the second sample data is greater than the data volume of the first sample data.

[0063] As described above, the second sample data is used to jointly train the learning hyperparameters of the connection layer and the visual feature layer. Therefore, the amount of the second sample data needs to be larger than that of the first sample data to ensure the training effect.

[0064] The second sample data includes second image sample data, second video sample data, second graphic sample data, second video description sample data, and second image text recognition data. Figure 7 As shown, the second image sample data and the second video sample data are input into the visual feature layer, where visual information is extracted from the second image sample data and the second video sample data. The second graphic and text sample data and the second image text recognition data are input into the multimodal feature layer, where text semantics are recognized in combination with the graphic and text sample data and the instructions corresponding to the image and text data. The visual features extracted from the visual feature layer are mapped to the multimodal feature layer via the connection layer to achieve fine-grained alignment of visual information and text semantics.

[0065] The second graphic sample data, the second video description sample data, and the second image text recognition data are matched according to a second preset ratio (pre-set data ratio). Specifically, the data volume of the second graphic sample data is greater than the data volume of the first graphic sample data, and the data volume of the second image text recognition data is greater than the data volume of the first image text recognition data; and the proportion of the second graphic sample data is greater than that of the first image text recognition data. Figure 5As shown, different sample data are extracted from different data sources. The total amount of the second graphic sample data, the second video description sample data and the second image text recognition data is 11M, of which the second graphic sample data (graphic caption data) accounts for 6.7M, the second image text recognition data (i.e., picture OCR data) accounts for 3.3M, and the second video description sample data (video caption data) accounts for 1M.

[0066] The third sample data includes third image sample data, third video sample data, and multimodal knowledge sample data, and the data volume of the third sample data is greater than the data volume of the second sample data.

[0067] As described above, the third sample data is used to jointly train the learning hyperparameters of the connection layer, visual feature layer and multimodal feature layer. Therefore, the data volume of the third sample data needs to be larger than the data volume of the first sample data and larger than the data volume of the second sample data to ensure the training effect.

[0068] The third sample data includes third image sample data, third video sample data and multimodal knowledge sample data, wherein the multimodal knowledge sample data covers sample data of multimodal data types such as caption data, image OCR data, mathematical knowledge math, text data text, question and answer data ShortQA for reading comprehension, and open source question and answer data OpenQA.

[0069] like Figure 8 As shown, the third image sample data and the third video sample data are used to be input into the visual feature layer, and the visual characteristic layer is used to extract visual information from the third image sample data and the third video sample data; the multimodal knowledge sample data is used to be input into the multimodal feature layer, and the knowledge injection training of the multimodal feature layer is performed in combination with the multimodal instruction set corresponding to the multimodal knowledge sample data; the visual features extracted by the visual feature layer are mapped to the multimodal feature layer through the connection layer, so as to extract effective visual features from the multimodal knowledge according to the visual information.

[0070] Among them, caption data, image OCR data, math data, text data, ShortQA data for reading comprehension, OpenQA data for open source questions and answers, and mixed data are proportioned according to the pre-set data ratio. Figure 5 As shown in the figure, different sample data are extracted from different data sources, with a total data volume of 36M, of which caption data accounts for 3.1M, image OCR data accounts for 9.6M, open source question answering data OpenQA accounts for 7.1M, question answering data ShortQA accounts for 0.9M, mathematical knowledge math accounts for 1.5M, text data accounts for 4.8M, and mixed data accounts for 9M.

[0071] In the above implementation, sample data with corresponding data ratios is used for training at different training stages, ensuring the training effect at each stage. Furthermore, different types of sample data are used for training at each stage to avoid the problem of a single data type leading to a discrepancy between the visual features extracted by the visual encoder and the feature space of the multimodal processing model.

[0072] In some optional embodiments, training the connection layer using the first sample data so that the pre-trained connection layer aligns the visual feature layer with the feature dimension of the multimodal processing model includes:

[0073] Step a1: Freeze the first parameter of the visual feature layer and the third parameter of the multimodal feature layer.

[0074] Step a2: extracting visual features from the first image sample data using the visual feature layer to obtain a first visual feature.

[0075] Step a3: extract features from the first image-text sample data and the first image-text recognition data using a multimodal feature layer to obtain first description features corresponding to the first image-text sample data and first recognition features corresponding to the first image-text recognition data.

[0076] Step a4: Use the first visual feature data to train the first parameter of the connection layer, and map the first visual feature to the multimodal feature layer through the connection layer so that the first visual feature is aligned with the first description feature and the first recognition feature.

[0077] The first parameter is the learning hyperparameter corresponding to the visual feature layer; the second parameter is the learning hyperparameter corresponding to the connection layer; and the third parameter is the learning hyperparameter corresponding to the multimodal feature layer.

[0078] During the training phase of the connection layer, the first parameter corresponding to the visual feature layer and the third parameter corresponding to the multimodal feature layer are frozen, and only the second parameter corresponding to the multilayer perceptron (MLP) of the connection layer is released.

[0079] Specifically, if Figure 6 As shown, the first image sample data is input into the visual feature layer, and visual features are extracted from the first image sample data according to the frozen first parameters to obtain corresponding first visual features.

[0080] The first image-text sample data and the first image text recognition data are input into the multimodal feature layer, and through the instructions corresponding to the first image-text sample data (such as "Please describe the content expressed by the above image"), the first image-text sample data is feature extracted according to the frozen third parameter to obtain the corresponding first description feature, and the visual content of the first image sample data is represented by the first description feature; through the instructions corresponding to the first image text recognition data (such as "Please identify the text content in the above image"), the first image text recognition data is feature extracted according to the frozen third parameter to obtain the corresponding first recognition feature, and the text content carried by the first image text recognition data is represented by the first recognition feature.

[0081] The second parameter is an adjustable parameter. The first visual feature output by the visual feature layer is used to adjust the second parameter corresponding to the connection layer, so as to achieve coarse-grained alignment of the visual feature layer and the multimodal feature layer in the feature dimension when mapping the first visual feature to the multimodal feature layer, that is, the first visual feature and the first descriptive feature are coarse-grained aligned in the feature dimension, and the first visual feature and the first identification feature are coarse-grained aligned in the feature dimension.

[0082] During the training phase of the connection layer, by pre-training the learning hyperparameters corresponding to the connection layer, the visual information extracted by the visual feature layer and the text semantics of the multimodal feature layer can be coarsely aligned in the feature dimension based on large-scale perception data.

[0083] In some optional embodiments, the visual feature layer is jointly trained using the second sample data and the pre-trained connection layer so that the visual feature layer extracts multimodal visual features, including:

[0084] Step b1: freeze the third parameter of the multimodal feature layer.

[0085] Step b2: extract visual features from the second image sample data and the second video sample data using the visual feature layer to obtain second picture visual features corresponding to the second image sample data and second video visual features corresponding to the second video sample data.

[0086] Step b3: extract features from the second image-text sample data and the second image-text recognition data using a multimodal feature layer to obtain second description features corresponding to the second image-text sample data and second recognition features corresponding to the second image-text recognition data.

[0087] Step b4, using the second image visual features, the second video visual features, the second description features and the second identification features, jointly train the second parameters of the connection layer and the first parameters of the visual feature layer, so that the visual feature layer performs multi-task pre-training and captures a variety of visual features that conform to the preset knowledge distribution.

[0088] During the joint training phase of the visual feature layer and the connection layer, the third parameter corresponding to the multimodal feature layer is frozen, and the second parameter corresponding to the multilayer perceptron (MLP) of the connection layer and the first parameter corresponding to the visual feature layer are released.

[0089] Specifically, if Figure 7 As shown, the second graphic sample data and the second image text recognition data are input into the multimodal feature layer, and through the instructions corresponding to the second graphic sample data (such as "Please describe the content expressed by the above image"), the second graphic sample data is feature extracted according to the frozen third parameter to obtain the corresponding second description feature, and the visual content of the second image sample data is represented by the second description feature; through the instructions corresponding to the second image text recognition data (such as "Please identify the text content in the above image"), the second image text recognition data is feature extracted according to the frozen third parameter to obtain the corresponding second recognition feature, and the text content carried by the first image text recognition data is represented by the first recognition feature.

[0090] The second image sample data and the second video sample data are input into the visual feature layer, and pre-training of the image visual feature extraction task and the video visual feature extraction task is performed according to the first parameters to obtain the second image visual features corresponding to the second image sample data and the second video visual features corresponding to the second video sample data. The second image visual features and the second video visual features are mapped to the multimodal feature layer according to the second parameters through the connection layer.

[0091] The first parameters of the visual feature layer and the second parameters of the connection layer are jointly trained in combination with the second image visual features, the second video visual features, the second description features and the second recognition features, so that when feature mapping is performed through the connection layer, fine-grained alignment of the multiple visual features captured by the visual feature layer and the text semantics of the multimodal feature layer in the feature space is achieved, that is, the semantics of visual elements that meet the preset knowledge distribution (that is, the real-world knowledge distribution) are captured through the pre-trained visual feature layer and the connection layer.

[0092] During the training phases of the connection layer and visual feature layer, the visual encoder's ability to extract diverse visual features is enhanced by pre-training the corresponding learning hyperparameters for the connection layer and visual feature layer. Furthermore, the introduction of additional video sample data for multi-task pre-training helps the visual encoder better capture the semantics of visual elements that are more closely aligned with real-world distributions, compared to the purely image sample data used during the connection layer training phase.

[0093] In some optional embodiments, the multimodal feature layer is jointly trained using the third sample data, the pre-trained connection layer, and the pre-trained visual feature layer to inject multimodal knowledge into the multimodal feature layer, including:

[0094] Step c1: extract visual features from the third image sample data and the third video sample data using a visual feature layer to obtain third picture visual features corresponding to the third image sample data and third video visual features corresponding to the third video sample data.

[0095] Step c2: extracting features from the multimodal knowledge sample data using the multimodal feature layer to obtain knowledge features corresponding to the multimodal knowledge sample data.

[0096] Step c3, using the third image visual features, the third video visual features and the knowledge features, jointly train the first parameters of the visual feature layer, the second parameters of the connection layer and the third parameters of the multimodal feature layer, so that the multimodal feature layer extracts multimodal visual features based on multimodal knowledge.

[0097] During the joint training phase of the visual feature layer, the connection layer, and the multimodal feature layer, the first parameter corresponding to the visual feature layer, the second parameter corresponding to the multilayer perceptron (MLP) of the connection layer, and the third parameter corresponding to the multimodal feature layer are released.

[0098] Specifically, if Figure 8 As shown, the third image sample data and the third video sample data are input into the visual feature layer, and pre-training of the image visual feature extraction task and the video visual feature extraction task is performed according to the first parameter to obtain the third image sample data corresponding to the third picture visual feature and the third video sample data corresponding to the third video visual feature, and through the connection layer, the third picture visual feature and the third video visual feature are mapped to the multimodal feature layer according to the second parameter.

[0099] Data of multimodal data types, such as caption data, image OCR data, mathematical knowledge (math), text data, ShortQA question-answering data for reading comprehension, OpenQA open source question-answering data, and mixed data, are input into the multimodal feature layer. Through the instructions of the corresponding data type, features of the data of each data type are extracted according to the third parameter to obtain the corresponding knowledge features. The text semantics of the multimodal knowledge sample data are represented by the knowledge features.

[0100] The first parameters of the visual feature layer, the second parameters of the connection layer, and the third parameters of the multimodal feature layer are jointly trained in combination with the third image visual features, the third video visual features, and the knowledge features. On the basis of achieving fine-grained alignment, more comprehensive world knowledge is injected into the multimodal feature layer, so that the trained multimodal feature layer can extract effective visual features from a variety of visual information inputs.

[0101] Therefore, the visual encoder generated after the above three stages of training can extract effective and robust visual features from diverse media data after being connected to various types of multimodal processing models trained based on the large language model architecture.

[0102] In some optional embodiments, for any one of the training stages of the visual feature layer, the connection layer, and the multimodal feature layer, a preset loss strategy is used to iteratively optimize the training stage until the training result meets the preset conditions.

[0103] A preset loss strategy is a pre-defined optimization loss strategy, such as cross-entropy loss. The preset condition indicates that the training results meet convergence and achieve the expected results. At any stage of training for the visual feature layer, connection layer, and multimodal feature layer, an autoregressive approach is used for training and inference, using the optimized loss predicted by the preset loss strategy. Iterations are performed based on this optimized loss until the training results meet convergence and achieve the expected prediction results.

[0104] Taking the cross entropy as the prediction optimization loss as an example, in each training stage, each prediction result predicts the next N+1 token based on the first N tokens of the input. During training, in order to complete the training of the entire sentence at one time, the lower triangular attention mask can be used to constrain the calculation of the self-attention layer, ensuring that only the previous features are used at each token, such as Figure 9 As shown in Figure 2. Data filtering is performed based on cross entropy loss to improve prediction results.

[0105] Among them, the cross entropy loss predicted using each token can be specifically expressed as:

[0106]

[0107] Among them, Loss represents the cross entropy loss value, which indicates the gap between the predicted result and the true label; n is the total number of token samples, that is, the number of predicted token samples; i represents the i-th predicted token; y represents the true token; x represents the probability of correct prediction, and ylog(x) represents the log-likelihood loss of the true label and the predicted probability.

[0108] By adopting a preset loss strategy to iteratively optimize the training process, we can ensure that the training effects of the visual feature layer, connection layer and multimodal feature layer meet the requirements of visual feature extraction, which is conducive to improving the effectiveness of visual encoder in extracting visual features.

[0109] As a specific application example of the disclosed embodiments, a visual encoder trained using the disclosed embodiments was used to generate corresponding target visual features. Simultaneously, a visual encoder from a related art was used to extract visual features from target media data to obtain the corresponding visual features. Comparison revealed that the visual encoder used in the disclosed embodiments outperformed the visual encoders from the related art in terms of effectiveness, robustness, and generalization.

[0110] Specifically, the visual encoder of the present disclosure is trained in a progressive manner based on the Qwen2.5 series model and the 1.5B scale model, including large and huge versions. Figure 10 As shown, after experimental verification, it can be found that the large version of the visual encoder disclosed in the present invention performs better on the open source opencompass list than open source models of the same scale, such as InternViT-300M and AIMv2-large; the visual encoder disclosed in the present invention has better robustness than the Internlm2.5 series and Qwen3 series; it has better performance at the 0.6B and 7B scales, which highlights its generalization.

[0111] In this embodiment, a visual feature extraction device is also provided, which is used to implement the above-mentioned embodiments and preferred embodiments. Details already described will not be repeated here. As used below, the term "module" may refer to a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation using hardware, or a combination of software and hardware, is also possible and contemplated.

[0112] This embodiment provides a visual feature extraction device, such as Figure 11 Shown, including:

[0113] The acquisition module 501 is used to acquire target media data.

[0114] Feature extraction module 502 is configured to input the target media data into the visual encoder of the multimodal processing model, and use the visual encoder to visually encode the target media data to obtain target visual features. The visual encoder is configured as a visual feature layer, a connection layer, and a multimodal feature layer. These layers are progressively trained based on a preset sequence.

[0115] In some optional implementations, the feature extraction module 502 includes:

[0116] The model training unit is used to progressively train the visual encoder based on a preset sequence. Specifically, the model training unit includes:

[0117] The first training subunit is used to train the connection layer using the first sample data, so that the pre-trained connection layer aligns the feature dimensions of the visual feature layer and the multimodal feature layer.

[0118] The second training subunit is used to jointly train the visual feature layer using the second sample data and the pre-trained connection layer, so that the visual feature layer extracts multimodal visual features.

[0119] The third training subunit is used to jointly train the multimodal feature layer using the third sample data, the pre-trained connection layer and the pre-trained visual feature layer, so as to inject multimodal knowledge into the multimodal feature layer.

[0120] In some optional implementations, the first sample data includes first image sample data, first graphic and text sample data, and first image and text recognition data, and the first graphic and text sample data and the first image and text recognition data are matched according to a first preset ratio.

[0121] The second sample data includes second image sample data, second video sample data, second graphic and text sample data, second video description sample data and second image text recognition data. The second graphic and text sample data, second video description sample data and second image text recognition data are matched according to a second preset ratio, and the data volume of the second sample data is greater than the data volume of the first sample data.

[0122] The third sample data includes third image sample data, third video sample data, and multimodal knowledge sample data, and the data volume of the third sample data is greater than the data volume of the second sample data.

[0123] In some optional embodiments, the first training subunit is specifically used to freeze the first parameter of the visual feature layer and the third parameter of the multimodal feature layer; use the visual feature layer to extract visual features of the first image sample data to obtain a first visual feature; use the multimodal feature layer to extract features of the first graphic sample data and the first image text recognition data to obtain a first descriptive feature corresponding to the first graphic sample data, and a first recognition feature corresponding to the first image text recognition data; use the first visual feature data to train the first parameter of the connection layer, and map the first visual feature to the multimodal feature layer through the connection layer to align the first visual feature with the first descriptive feature and the first recognition feature.

[0124] In some optional embodiments, the second training subunit is specifically used to freeze the third parameter of the multimodal feature layer; use the visual feature layer to extract visual features of the second image sample data and the second video sample data to obtain the second picture visual features corresponding to the second image sample data, and the second video visual features corresponding to the second video sample data; use the multimodal feature layer to extract features of the second graphic sample data and the second image text recognition data to obtain the second description features corresponding to the second graphic sample data, and the second recognition features corresponding to the second image text recognition data; use the second picture visual features, the second video visual features, the second description features and the second recognition features to jointly train the second parameters of the connection layer and the first parameters of the visual feature layer, so that the visual feature layer performs multi-task pre-training and captures a variety of visual features that conform to the preset knowledge distribution.

[0125] In some optional embodiments, the third training subunit is specifically used to use the visual feature layer to extract visual features of the third image sample data and the third video sample data to obtain third picture visual features corresponding to the third image sample data, and third video visual features corresponding to the third video sample data; use the multimodal feature layer to extract features of the multimodal knowledge sample data to obtain knowledge features corresponding to the multimodal knowledge sample data; use the third picture visual features, the third video visual features and the knowledge features to jointly train the first parameters of the visual feature layer, the second parameters of the connection layer and the third parameters of the multimodal feature layer, so that the multimodal feature layer extracts multimodal visual features based on multimodal knowledge.

[0126] In some optional implementations, the model training unit may further include:

[0127] The iterative optimization module is used to iteratively optimize the training stage of any training stage in the visual feature layer, connection layer and multimodal feature layer using a preset loss strategy until the training result meets the preset conditions.

[0128] The visual feature extraction device provided in the embodiments of the present disclosure can execute the visual feature extraction method provided in any embodiment of the present disclosure, and has the functional modules and beneficial effects corresponding to the execution method. By progressively training the visual encoder configured as a visual feature layer, a connection layer and a multimodal feature layer in a preset order, the visual features that can be extracted by the visual encoder can be gradually aligned with the feature space of the multimodal processing model through progressive training. Then, the target media data is visually encoded according to the trained visual encoder, and the obtained target visual features can be aligned with the feature space of the multimodal processing model faster and better, thereby improving the visual understanding ability of the multimodal processing model. The further functional description of each of the above modules and units is the same as that of the corresponding embodiment above, and will not be repeated here.

[0129] Figure 12 A schematic structural diagram of an electronic device provided in an embodiment of the present disclosure.

[0130] The following specific reference Figure 12 , which shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present disclosure. The electronic device may include a processor (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a memory 608 into a random access memory (RAM) 603. Various programs and data required for the operation of the electronic device are also stored in the RAM 603. The processor 601, ROM 602, and RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0131] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a memory 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 12 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all of the devices shown, and more or fewer devices may be implemented or possessed instead.

[0132] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 609, or installed from the memory 608, or installed from the ROM 602. When the computer program is executed by the processor 601, the above-mentioned functions defined in the method for extracting visual features of the embodiment of the present disclosure are performed.

[0133] Figure 12 The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0134] The embodiments of the present disclosure also provide a computer-readable storage medium. The above-mentioned method according to the embodiments of the present disclosure can be implemented in hardware, firmware, or implemented as a computer code that can be recorded in a storage medium, or implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and downloaded through a network and will be stored in a local storage medium, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state drive, etc.; further, the storage medium can also include a combination of the above-mentioned types of memory. It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor or hardware, the method for extracting visual features shown in the above embodiment is implemented.

[0135] A portion of the present disclosure may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present disclosure through the operation of the computer. Those skilled in the art should understand that the form in which the computer program instruction exists in a computer-readable medium includes but is not limited to a source file, an executable file, an installation package file, etc. Accordingly, the way in which the computer program instruction is executed by the computer includes but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium that can be accessed by the computer.

[0136] Although the embodiments of the present disclosure have been described with reference to the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A method for extracting visual features, characterized in that: The method comprises: Get target media data; Visually encoding the target media data using a visual encoder in a multimodal processing model to obtain target visual features; The visual encoder is configured as a visual feature layer, a connection layer and a multimodal feature layer, and the visual feature layer, the connection layer and the multimodal feature layer are obtained by progressive training based on a preset order.

2. The method according to claim 1, characterized in that The visual encoder is progressively trained based on a preset sequence, comprising: Training the connection layer using the first sample data so that the pre-trained connection layer aligns feature dimensions of the visual feature layer and the multimodal feature layer; Using the second sample data and the pre-trained connection layer, jointly training the visual feature layer so that the visual feature layer extracts multimodal visual features; The multimodal feature layer is jointly trained using the third sample data, the pre-trained connection layer, and the pre-trained visual feature layer to inject multimodal knowledge into the multimodal feature layer.

3. The method according to claim 2, characterized in that The first sample data includes first image sample data, first graphic and text sample data, and first image and text recognition data, and the first graphic and text sample data and the first image and text recognition data are matched according to a first preset ratio; The second sample data includes second image sample data, second video sample data, second graphic sample data, second video description sample data, and second image text recognition data, wherein the second graphic sample data, the second video description sample data, and the second image text recognition data are matched according to a second preset ratio, and the data volume of the second sample data is greater than the data volume of the first sample data; The third sample data includes third image sample data, third video sample data, and multimodal knowledge sample data, and the data volume of the third sample data is greater than the data volume of the second sample data.

4. The method according to claim 3, characterized in that The step of training the connection layer using the first sample data so that the pre-trained connection layer aligns the visual feature layer with the feature dimension of the multimodal processing model includes: Freezing a first parameter of the visual feature layer and a third parameter of the multimodal feature layer; Extracting visual features from the first image sample data using the visual feature layer to obtain a first visual feature; Performing feature extraction on the first image-text sample data and the first image-text recognition data using the multimodal feature layer to obtain a first descriptive feature corresponding to the first image-text sample data and a first recognition feature corresponding to the first image-text recognition data; The first visual feature is used to train a second parameter of the connection layer, and the first visual feature is mapped to the multimodal feature layer through the connection layer, so that the first visual feature is aligned with the first description feature and the first recognition feature.

5. The method according to claim 3, characterized in that The method of jointly training the visual feature layer using the second sample data and the pre-trained connection layer so that the visual feature layer extracts multimodal visual features includes: Freezing a third parameter of the multimodal feature layer; Performing visual feature extraction on the second image sample data and the second video sample data using the visual feature layer to obtain second picture visual features corresponding to the second image sample data and second video visual features corresponding to the second video sample data; Performing feature extraction on the second image-text sample data and the second image-text recognition data using the multimodal feature layer to obtain a second descriptive feature corresponding to the second image-text sample data and a second recognition feature corresponding to the second image-text recognition data; The second parameters of the connection layer and the first parameters of the visual feature layer are jointly trained using the second image visual features, the second video visual features, the second description features, and the second identification features, so that the visual feature layer performs multi-task pre-training and captures a variety of visual features that conform to the preset knowledge distribution.

6. The method according to claim 3, characterized in that The method of jointly training the multimodal feature layer using the third sample data, the pre-trained connection layer, and the pre-trained visual feature layer to inject multimodal knowledge into the multimodal feature layer includes: Performing visual feature extraction on the third image sample data and the third video sample data using the visual feature layer to obtain third picture visual features corresponding to the third image sample data and third video visual features corresponding to the third video sample data; Performing feature extraction on the multimodal knowledge sample data using the multimodal feature layer to obtain knowledge features corresponding to the multimodal knowledge sample data; The first parameter of the visual feature layer, the second parameter of the connection layer and the third parameter of the multimodal feature layer are jointly trained using the third image visual feature, the third video visual feature and the knowledge feature, so that the multimodal feature layer extracts multimodal visual features based on multimodal knowledge.

7. The method according to any one of claims 4 to 6, characterized in that Also includes: For any one training stage of the visual feature layer, the connection layer and the multimodal feature layer, a preset loss strategy is used to iteratively optimize the training stage until the training result meets the preset conditions.

8. A visual feature extraction device, characterized in that: The device comprises: An acquisition module, used to acquire target media data; a feature extraction module, configured to input the target media data into a visual encoder of a multimodal processing model, and perform visual encoding on the target media data using the visual encoder to obtain target visual features; The visual encoder is configured as a visual feature layer, a connection layer and a multimodal feature layer, and the visual feature layer, the connection layer and the multimodal feature layer are obtained by progressive training based on a preset order.

9. An electronic device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the visual feature extraction method according to any one of claims 1 to 7 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the visual feature extraction method according to any one of claims 1 to 7.

11. A computer program product, characterized in that The method comprises computer instructions for causing a computer to execute the method for extracting visual features according to any one of claims 1 to 7.

Images