Data processing method, device, electronic device and storage medium

Abstract

The present invention discloses a data processing method, device, electronic device and storage medium, the method comprising: obtaining text features of a text to be processed and image features of an image to be processed; the text to be processed is a description text corresponding to the image to be processed; based on the text features, key image features in the image features are determined; the key image features are image features with corresponding semantic information in the text features; based on the text features and the key image features, multimodal features are obtained by fusion processing, the multimodal features are encoded into a preset semantic space, and a multimodal feature encoding vector is obtained; based on the multimodal feature encoding vector, sequence annotation processing is performed to obtain a target object label sequence; the target object label sequence is used to perform structured processing on the text to be processed to obtain target structured data. The present invention improves the accuracy of the text structured processing results.

Description

Technical Field

[0001] The present invention relates to the field of Internet technology, and in particular to a data processing method, device, electronic equipment and storage medium. Background Art

[0002] With the development of Internet technology, more and more information is displayed in multiple modalities, including visual (such as images) and text. In some application scenarios, unstructured text descriptions are often converted into structured data, such as "attribute-attribute value", based on text and corresponding images, for subsequent content understanding systems such as question-answering and recommendation.

[0003] In the related art, for text and the image corresponding to the text, the image information and the text information are usually simply fused directly without considering the language information such as color, shape, size, etc. represented by the image information. It is impossible to align the language information represented in the image information with the text features with the same meaning, which reduces the fusion effect and leads to poor accuracy of the text structuring processing results in the related art. Summary of the invention

[0004] In order to solve the problems of the prior art, the embodiments of the present invention provide a data processing method, device, electronic device and storage medium. The technical solution is as follows:

[0005] In one aspect, a data processing method is provided, the method comprising:

[0006] Acquire text features of a text to be processed and image features of an image to be processed; the text to be processed is a description text corresponding to the image to be processed;

[0007] Determine a key image feature in the image feature based on the text feature; the key image feature is an image feature having corresponding semantic information in the text feature;

[0008] Based on the text feature and the key image feature, a fusion process is performed to obtain a multimodal feature, and the multimodal feature is encoded into a preset semantic space to obtain a multimodal feature encoding vector;

[0009] Sequence labeling processing is performed based on the multimodal feature encoding vector to obtain a target object label sequence; the target object label sequence is used to perform structured processing on the text to be processed to obtain target structured data.

[0010] In another aspect, a data processing device is provided, the device comprising:

[0011] A data acquisition module, used to acquire text features of a text to be processed and image features of an image to be processed; the text to be processed is a description text corresponding to the image to be processed;

[0012] A key image feature determination module, used to determine a key image feature in the image features based on the text features; the key image feature is an image feature having corresponding semantic information in the text features;

[0013] A fusion module, configured to obtain a multimodal feature by fusing the text feature and the key image feature, and encode the multimodal feature into a preset semantic space to obtain a multimodal feature encoding vector;

[0014] A sequence labeling module is used to perform sequence labeling processing based on the multimodal feature encoding vector to obtain a target object label sequence; the target object label sequence is used to perform structured processing on the text to be processed to obtain target structured data.

[0015] In an exemplary embodiment, the key image feature determination module includes:

[0016] A cross attention weight determination module, used for determining a cross attention weight based on the text feature and the image feature;

[0017] The key image feature determination submodule is used to adjust the image features based on the cross attention weights to obtain key image features.

[0018] In an exemplary embodiment, the fusion module includes:

[0019] A self-attention weight determination module, used to determine the self-attention weight based on the text feature;

[0020] A key text feature determination module, used for adjusting the text feature based on the self-attention weight to obtain a key text feature;

[0021] The fusion submodule is used to fuse the text features, the key image features and the key text features to obtain multimodal features.

[0022] In an exemplary embodiment, the sequence tagging module includes:

[0023] A first target preset object determination module, used to determine a first target preset object among a plurality of preset objects based on the multimodal feature encoding vector; the first target preset object is a preset object existing in the text to be processed;

[0024] A first determination module is used to determine a first target object prototype vector corresponding to the first target preset object in an object prototype vector set; the object prototype vector set includes an object prototype vector of each preset object in the plurality of preset objects;

[0025] A first matching processing module, configured to perform matching processing on the multimodal feature coding vector based on the first target object prototype vector to obtain a target multimodal feature coding vector corresponding to the first target preset object;

[0026] The sequence labeling submodule is used to perform sequence labeling processing based on the target multimodal feature encoding vector to obtain a target object label sequence corresponding to the first target preset object.

[0027] In an exemplary embodiment, the first target preset object determination module includes:

[0028] A classification feature coding determination module is used to extract a feature coding vector at a location of a preset classification identifier from the multimodal feature coding vector to obtain a classification feature coding vector; the preset classification identifier is an identifier added to the beginning of the text to be processed when generating the text feature;

[0029] A first classification processing module is used to perform classification processing based on the classification feature coding vector to obtain a classification result; the classification result includes a confidence level corresponding to each preset object in a plurality of preset objects, and the confidence level represents the probability that the corresponding preset object exists in the text to be processed;

[0030] The second determination module is used to determine a first target preset object among the multiple preset objects whose confidence level is greater than a confidence threshold.

[0031] In an exemplary embodiment, the first matching processing module includes:

[0032] A matching degree determination module, used to determine the matching degree between each feature coding vector in the multimodal feature coding vector and the first target object prototype vector;

[0033] The splicing processing module is used to determine the product of the matching degree and the corresponding feature vector, and splice the result of the product with the corresponding feature vector in the channel dimension to obtain a target multimodal feature coding vector.

[0034] In an exemplary embodiment, the apparatus further comprises a training module, wherein the training module comprises:

[0035] A sample data acquisition module is used to acquire sample data in a sample data set, wherein the sample data includes sample text and corresponding sample images, and the sample text corresponds to a reference object label sequence;

[0036] A sample multimodal feature encoding module, used for inputting the sample data into a feature encoder of an initial data processing model to obtain an output sample multimodal feature encoding vector; the feature encoder is used for determining a sample text feature corresponding to the sample text based on a pre-trained word vector layer and a sample image feature corresponding to the sample image based on a pre-trained visual model, determining a sample key image feature in the sample image feature based on the sample text feature, performing a fusion process based on the sample text feature and the sample key image feature to obtain a sample multimodal feature, and performing an encoding process on the sample multimodal feature based on a pre-trained language model to obtain the sample multimodal feature encoding vector;

[0037] A sequence probability prediction module, used for inputting the sample multimodal feature encoding vector into the feature decoder of the initial data processing model for sequence labeling processing to obtain prediction probabilities of multiple prediction object label sequences; the multiple prediction object label sequences include the reference object label sequence;

[0038] A loss value determination module, used to determine a loss value based on the prediction probabilities of the plurality of prediction object label sequences;

[0039] A parameter updating training module is used to update the model parameters in the initial data processing model based on the loss value until the training end condition is met to obtain the data processing model.

[0040] In an exemplary embodiment, the parameter update training module is specifically configured to update the model parameters in the feature encoder and the feature decoder respectively based on the loss value.

[0041] In an exemplary embodiment, the parameter updating training module includes:

[0042] A first update amplitude determination module, configured to determine a first update amplitude of a first model parameter based on the loss value; the first model parameter is a model parameter in the pre-trained visual model;

[0043] A second update amplitude determination module is used to determine a second update amplitude of a second model parameter based on the loss value, wherein the second update amplitude is greater than the first update amplitude; the second model parameter includes a model parameter in the feature encoder except the first model parameter and a model parameter in the feature decoder;

[0044] An updating submodule is used to update corresponding model parameters based on the first updating amplitude and the second updating amplitude respectively.

[0045] In an exemplary embodiment, the sequence probability prediction module includes:

[0046] An object range prediction module, used for inputting the sample multimodal feature encoding vector into the feature decoder to perform object range prediction, and obtain an object range prediction result; the object range prediction result includes a second target preset object among a plurality of preset objects;

[0047] A third determining module is used to determine a second target object prototype vector corresponding to the second target preset object in the object prototype vector set;

[0048] A second matching processing module, used for performing matching processing on the sample multimodal feature coding vector based on the second target object prototype vector to obtain a target sample multimodal feature coding vector corresponding to the second target preset object;

[0049] The sequence probability prediction submodule is used to input the target sample multimodal feature encoding vector into the sequence labeling model of the feature decoder for sequence labeling, so as to obtain the prediction probability of multiple predicted object label sequences corresponding to the second target preset object.

[0050] In an exemplary embodiment, the object range prediction module includes:

[0051] A sample classification feature coding extraction module is used to extract a feature coding vector at a location of a preset classification identifier from the sample multimodal feature coding vector to obtain a sample classification feature coding vector; the preset classification identifier is an identifier added to the beginning of the sample text when generating the sample text feature;

[0052] A second classification processing module is used to input the sample classification feature encoding vector into the classification network of the feature decoder for classification processing to obtain a prediction classification result; the prediction classification result includes a prediction confidence corresponding to each preset object in a plurality of preset objects, and the prediction confidence represents the probability that the corresponding preset object exists in the sample text;

[0053] The fourth determination module is used to determine a second target preset object among the multiple preset objects, the prediction confidence of which is greater than a confidence threshold.

[0054] In an exemplary embodiment, the loss value determination module includes:

[0055] A first loss value determination module, configured to determine a first loss value based on the prediction probabilities of the plurality of prediction object label sequences corresponding to the second target preset object through a first loss function;

[0056] A second loss value determination module, configured to determine a second loss value based on the predicted classification result and reference classification label information corresponding to the sample text by using a second loss function; the reference classification label information indicates whether each of the plurality of preset objects exists in the sample text;

[0057] A total loss value determination module is used to determine a total loss value based on the first loss value and the second loss value.

[0058] In an exemplary embodiment, the first loss value determination module includes:

[0059] A first determination submodule is configured to, when the second target preset object matches the reference classification label information, use the predicted probability corresponding to the reference object label sequence in the multiple predicted object label sequences as the maximization target of the first loss function, and determine a first loss value using the first loss function based on the maximization target and the predicted probability of the multiple predicted object label sequences;

[0060] The second determination submodule is used to use the prediction probability corresponding to the target prediction object label sequence in the multiple prediction object label sequences as the maximization target of the first loss function when the second target preset object does not match the reference classification label information, and determine the first loss value using the first loss function based on the maximization target and the prediction probability of the multiple prediction object label sequences; each object label in the target prediction object label sequence represents a non-preset object.

[0061] In an exemplary embodiment, the training module further includes:

[0062] A fifth determination module, configured to determine the object corresponding to each sample text based on a reference object label sequence corresponding to each sample text in the sample data set;

[0063] A preset object set determination module, used to generate a preset object set corresponding to the sample data set based on the object corresponding to each of the sample texts;

[0064] The reference classification label information generating module is used to generate reference classification label information corresponding to the sample text based on the preset object set and the object corresponding to the sample text in each sample data.

[0065] On the other hand, an electronic device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement any of the data processing methods described above.

[0066] On the other hand, a computer-readable storage medium is provided, in which at least one instruction or at least one program is stored, and the at least one instruction or the at least one program is loaded and executed by a processor to implement a data processing method as described in any of the above aspects.

[0067] On the other hand, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device performs any of the above-mentioned data processing methods.

[0068] The embodiment of the present invention determines key image features in image features that have corresponding semantic information in text features based on text features, obtains multimodal features based on the fusion of text features and key image features, encodes the multimodal features into a preset semantic space to obtain a multimodal feature encoding vector, thereby aligning language information such as color, shape, size, etc. represented in the image with text features of the same meaning, achieving better fusion encoding, and subsequently improving the accuracy of text structuring results when sequence annotation processing is performed based on the multimodal feature encoding vector and structured processing is performed on the text to be processed based on the target object label sequence. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0070] Figure 1 It is a flowchart of a data processing method provided by an embodiment of the present invention;

[0071] Figure 2 is a schematic diagram of feature processing based on text guidance provided by an embodiment of the present invention;

[0072] Figure 3a It is a schematic diagram of a process of performing sequence labeling processing based on a multimodal feature coding vector to obtain a target object label sequence provided by an embodiment of the present invention;

[0073] Figure 3b It is a schematic diagram of a process of determining a first target preset object among multiple preset objects based on a multimodal feature coding vector provided by an embodiment of the present invention;

[0074] Figure 4 This is an example of selecting a first target preset attribute from a plurality of preset attributes for an input text and a corresponding input image provided by an embodiment of the present invention;

[0075] Figure 5a is a structural diagram of a data processing model provided by an embodiment of the present invention;

[0076] Figure 5b is a structural schematic diagram of another data processing model provided by an embodiment of the present invention;

[0077] Figure 6 is a structural schematic diagram of another data processing model provided by an embodiment of the present invention;

[0078] Figure 7 is a schematic diagram of another data processing method provided by an embodiment of the present invention;

[0079] Figure 8 is a structural block diagram of a data processing device provided by an embodiment of the present invention;

[0080] Fig. 9 The present invention provides a hardware structure block diagram of an electronic device. DETAILED DESCRIPTION

[0081] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0082] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0083] It is understandable that in the specific implementation of this application, related data such as user information is involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0084] The following is an explanation of the terms involved in the embodiments of the present invention.

[0085] BiLSTM: Bi-directional Long Short Term Memory, a bidirectional long short-term memory network, is composed of a forward long short-term memory network and a backward long short-term memory network, and can be a time recursive neural network.

[0086] CRF: Conditional Random Field, a conditional random field model, can be a conditional probability distribution model of another set of output random variables under a given set of input random variables. Its characteristic is that it assumes that the output random variables constitute a Markov random field.

[0087] MLP: Multiple Layer Perceptron, a feedforward neural network model that maps multiple input data sets to a single output data set.

[0088] BERT: Bidirectional Encoder Representation from Transformers, a pre-trained language model.

[0089] ResNet: Residual Network, residual neural network, each node learns the residual instead of the parameter itself.

[0090] In the related art, images are usually used as auxiliary information of texts. The image information and text information are simply fused and then sequence annotation is performed to obtain a result sequence. The text is structured with the result sequence to obtain structured data, where the structured processing may include attribute value extraction, named entity recognition, relationship extraction, form filling, etc. Since the above-mentioned simple fusion does not take into account the language information such as color, shape, size represented by the image information, it is impossible to align the language information represented in the image information with the text features of the same meaning, which reduces the fusion effect and leads to poor accuracy of the text structured processing results in the related art.

[0091] In view of this, an embodiment of the present invention proposes a data processing method, which determines key image features in image features based on text features, where the key image features are image features that have corresponding semantic information in the text features, and fuses the text features and the key image features, and encodes the fused multimodal features into a preset semantic space to obtain a multimodal feature encoding vector, thereby aligning the language information such as color, shape, size represented in the image with the text features of the same meaning, achieving better fusion encoding, and then subsequently performing sequence annotation processing based on the multimodal feature encoding vector and performing structured processing on the text to be processed based on the target object label sequence, thereby greatly improving the accuracy of the text structuring processing results.

[0092] The embodiments of the present invention can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, etc.

[0093] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making. Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level technology and software-level technology. The basic technologies of artificial intelligence generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, and mechatronics. Artificial intelligence software technology mainly includes several major directions such as computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0094] Among them, Computer Vision (CV) is a science that studies how to make machines "see". To put it more specifically, it refers to machine vision such as using cameras and computers to replace human eyes to identify and measure targets, and further perform graphic processing to make computer processing into images that are more suitable for human eye observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multidimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and map construction, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0095] Natural language processing (NLP) is an important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that can achieve effective communication between people and computers using natural language. Natural language processing is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field will involve natural language, that is, the language people use in daily life, so it is closely related to the study of linguistics. Natural language processing technology usually includes text processing, semantic understanding, machine translation, robot question answering, knowledge graph and other technologies.

[0096] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0097] The data processing method of the embodiment of the present invention can be used for a data processing device, and the data processing device can be configured in an electronic device, and the electronic device can be a terminal or a server. Among them, the terminal can include but is not limited to mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle terminals, aircraft, etc. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network, content distribution network) and big data and artificial intelligence platforms.

[0098] In an exemplary embodiment, all electronic devices can be node devices in the blockchain system, and can share the acquired and generated information with other node devices in the blockchain system, thereby realizing information sharing among multiple node devices. Multiple node devices in the blockchain system can be configured with the same blockchain, which is composed of multiple blocks, and the adjacent blocks have an association relationship, so that when the data in any block is tampered, it can be detected by the next block, thereby preventing the data in the blockchain from being tampered with, and ensuring the security and reliability of the data in the blockchain.

[0099] In practical applications, the data processing method of the embodiment of the present invention can be used in commodity application scenarios such as commodity labeling, commodity entry, commodity retrieval, and commodity recommendation. The specific commodity application scenarios are described as follows:

[0100] Product labeling and entry: Labeled data is very important for the training of algorithm models. The embodiments of the present invention can assist manual labeling in labeling scenarios, improve labeling efficiency, and reduce labor costs. Similarly, in e-commerce product entry scenarios, it can assist manual labeling of defined attributes to improve entry efficiency.

[0101] Product retrieval: When users search for products on a shopping platform, they generally add key attribute words, such as product words, brands, styles, colors, price ranges, etc. The technical solution of the embodiment of the present invention can automatically identify the attributes of the product based on the title and picture of the product, thereby retrieving matching products and outputting them to the user, thereby improving the retrieval performance.

[0102] Product recommendation: Assuming that user description information is available, the attribute features of the product can be extracted using the embodiment of the present invention to calculate whether the user and the product are related, provide the user with more personalized and refined recommendations, and improve the recommendation effect.

[0103] It should be noted that the above is an exemplary description of the available application scenarios of the embodiments of the present invention. In actual applications, the use scenarios of the embodiments of the present invention are not limited to the above-mentioned commodity application scenarios, but can also be applied to more general natural language processing / computer vision multimodal feature fusion scenarios, such as image and text retrieval, etc. The embodiments of the present invention do not specifically limit the applicable application scenarios.

[0104] See also Figure 1 , which shows a flow chart of a data processing method provided by an embodiment of the present invention. It should be noted that this specification provides method operation steps as described in the embodiment or flow chart, but may include more or fewer operation steps based on conventional or non-creative labor. The order of steps listed in the embodiment is only one way of executing the steps among many orders, and does not represent the only order of execution. When the system or product is executed in practice, it can be executed in sequence or in parallel according to the method shown in the embodiment or the accompanying drawings (for example, in a parallel processor or multi-threaded processing environment). Specifically, Figure 1 As shown, the method may include:

[0105] S101, obtaining text features of a text to be processed and image features of an image to be processed.

[0106] The text to be processed is the description text corresponding to the image to be processed. For example, when the image to be processed is a product picture, the text to be processed may be the product title information.

[0107] Specifically, the text feature of the text to be processed may be a word vector sequence obtained by embedding the text to be processed into word vectors.

[0108] In the process of embedding word vectors, each word or Chinese character in the text to be processed can be tokenized (i.e., tokenized) through a preset dictionary and converted into a fixed ID (Identity), and then the corresponding word vector is found in the word vector space according to the ID, thereby converting the text to be processed into a corresponding word vector sequence. The word vector space can be a pre-trained vector space, and the training of the word vector space will be described in detail in the subsequent content of the embodiment of the present invention.

[0109] Specifically, the image feature of the image to be processed may be a feature vector sequence obtained by extracting features from the image to be processed, and the number of channels of the feature vector in the feature vector sequence is the same as the number of channels of the above-mentioned word vector.

[0110] In feature extraction, the image to be processed can be processed by a multi-layer convolutional neural network (CNN), and the image to be processed can be converted into a feature block of a specific size (such as a spatial length and width of 7x7) and a specific number of channels (such as 1024). The feature block is mapped to the same number of channels as the word vector (such as 768) through MLP.

[0111] S103: Determine key image features among the image features based on the text features.

[0112] The key image feature is an image feature having corresponding semantic information in the text feature, thereby achieving an effect of selecting the key area information corresponding to the text in the image.

[0113] In an exemplary embodiment, the key image features can be obtained by processing the text features and the image features based on the cross-attention mechanism, which is a type of attention mechanism. In order to facilitate understanding of the application of the cross-attention mechanism in the embodiment of the present invention, the attention mechanism is described below.

[0114] Given two input feature sequences The attention mechanism uses three weight matrices Map them linearly to in, represents a vector or tensor in the real number field, S1 and S2 are the lengths of the two sequences, D and D′ are the channel dimensions of the feature sequence and the feature sequence inside the attention mechanism, respectively. The linear mapping formula is shown in the following formula (1):

[0115] Q = x Q W Q , K = x K W K , V = x V W V (1)

[0116] Among them, Q represents a "query" to obtain the part of V that is useful to Q. Since the sources of the two may be different, such as features extracted from text and images respectively, it is inappropriate to directly calculate the similarity between the two to represent "whether it is useful". Therefore, K and V constitute a "key-value pair", and K represents the "key" of each position in V, which replaces V to calculate the similarity with Q. The calculation formula of the attention mechanism is shown in formula (2):

[0117]

[0118] By transposing and multiplying Q and K in equation (2), the attention map represents the degree of match between each position of Q and K, and is normalized by the number of channel dimensions. Based on this mapping, for each position of Q, all positions in V are multiplied by a specific match score and added together to obtain an attention output with the same shape as Q itself. A represents the useful information that each position of Q obtains from V, similar to how humans only pay more attention to useful information.

[0119] Based on this, when determining the key image features in the image features based on the text features, the above step S103 may include:

[0120] Determine cross-attention weights based on text features and image features;

[0121] The image features are adjusted based on the above cross-attention weights to obtain key image features.

[0122] Specifically, in order to implement the text-dominated cross-attention mechanism, the embodiment of the present invention assumes that the “query” input is the text feature x Q =x t , set the "key-value pair" input to the image feature x K =x V =x v , then the key image features can be obtained by the following formula (3):

[0123]

[0124] Among them, x t represents text features, x v Represents image features; CrossA(x t ,x v ) characterize key image features; represents the cross attention weight.

[0125] Considering that the dimension of the key image features obtained based on the attention mechanism is the dimension of the feature sequence inside the attention mechanism, in order to make the dimension of the key image features consistent with the dimension of the text features for the convenience of subsequent fusion processing, after the key image features are obtained based on the attention mechanism, a simple MLP can be used to remap the key image features from the feature sequence dimension inside the attention mechanism back to the input feature sequence dimension, that is, the final key image features can be expressed as CrossAtt(x t ,x v )W m ,in, is the weight matrix of MLP, which can be obtained through pre-training.

[0126] In the above implementation, through the cross-attention mechanism, with text features as "queries" and image features as "key-value pairs", key image features that have corresponding semantic information in text features are extracted from image features.

[0127] S105, performing fusion processing based on the text feature and the key image feature to obtain a multimodal feature, encoding the multimodal feature into a preset semantic space, and obtaining a multimodal feature encoding vector.

[0128] The fusion process may be to splice text features and key image features.

[0129] When encoding multimodal features into a preset semantic space, the input multimodal features can be encoded based on a pre-trained large language model, thereby achieving the joint encoding of language-related image information and text information into a unified semantic space.

[0130] It can be understood that the multimodal feature coding vector includes feature coding vectors corresponding to each position of the text to be processed, that is, the multimodal feature coding vector is essentially a sequence of feature coding vectors corresponding to each position of the text to be processed.

[0131] In an exemplary embodiment, in order to further increase the proportion of useful information in the multimodal features to improve the accuracy of subsequent processing results, key text information can also be extracted from the text features based on the self-attention mechanism, and then the text features, key text features and key image features are fused during the fusion process to obtain the multimodal features. Based on this, the above step S105 may include:

[0132] Determining a self-attention weight based on the text feature;

[0133] Adjusting the text feature based on the self-attention weight to obtain a key text feature;

[0134] The text features, the key image features and the key text features are fused to obtain multimodal features.

[0135] Specifically, in order to implement the text-dominated self-attention mechanism, the embodiment of the present invention assumes that the “query” input is the text feature x Q =x t , set the "key-value pair" input to the text feature x K =x V =x t , then the key text features can be obtained by the following formula (4):

[0136]

[0137] Among them, x t Represents text features; SelfAtt(x t ) characterizes key text features; represents the self-attention weight.

[0138] Considering that the dimension of the key text feature obtained based on the attention mechanism is the dimension of the feature sequence inside the attention mechanism, in order to make the dimension of the key text feature consistent with the dimension of the text feature for subsequent fusion processing, after the key text feature is obtained based on the attention mechanism, a simple MLP can be used to remap the key text feature from the dimension of the feature sequence inside the attention mechanism back to the dimension of the input feature sequence, that is, the final key text feature can be expressed as SelfAtt(x t )W m ,in, is the weight matrix of MLP, which can be obtained through pre-training.

[0139] Therefore, the text-guided multimodal feature E m It can be expressed as the following formula (5):

[0140] E m =x t +(SelfAtt(x t )+CrossAtt(x t , x v ))W m (5)

[0141] In the specific implementation, the maximum length S1 of the sequence of text features is generally 64 for Chinese data and 128 for English data. If it is insufficient, it is supplemented with a blank symbol "[PAD]". The sequence length S2 of image features is fixed to 49, and the corresponding input image size is scaled to 224x224. The feature channel dimension D is 768, and the internal sequence dimension D' of attention is 200.

[0142] In the above implementation, when fusing text features and key image features, key text features obtained based on the self-attention mechanism are further considered, so as to minimize the scope of modal information guided by the text, which is beneficial to improving the accuracy of subsequent processing results.

[0143] like Figure 2 As shown in the figure, the text to be processed is the description information of the product, and the image to be processed is the image corresponding to the product. After text-guided self-attention and cross-attention processing, the key text features "square collar, short sleeves, long skirt, floral print" and the corresponding key image features in the image can be obtained. The text features, key text features and key image features are fused and encoded into the same semantic space using a large language model.

[0144] S107, performing sequence labeling processing based on the multimodal feature encoding vector to obtain a target object label sequence.

[0145] The target object label sequence is used to perform structured processing on the text to be processed to obtain target structured data.

[0146] Specifically, the object labels in the target object label sequence are associated with the actual structured processing tasks. For example, when the structured processing task is attribute value extraction, the object label can be an attribute label; when the structured processing task is named entity recognition, the object label can be a named entity label; when the structured processing task is relationship extraction, the object label can be a relationship label.

[0147] In practical applications, the target object label sequence is a result sequence based on a sequence annotation format, which can be composed of three symbols: "B, I, O". B stands for Begin, which means the beginning; I stands for Intermediate, which means the middle; O stands for Other, which is used to mark irrelevant characters. In order to distinguish the object information of different objects, the object identifier is usually marked after the above three symbols. Taking the attribute value extraction task as an example, the attribute name is usually marked after "B" or "I", such as "B-battery power", "I-sleeve length", etc.

[0148] In a specific implementation, the text substrings corresponding to the character B representing the beginning and the character I representing the middle in the text to be processed can be extracted, and then the target structured data can be obtained based on the corresponding objects and the extracted text information.

[0149] In the above embodiment, key image features that have corresponding semantic information in the text features are determined in the image features based on the text features, and multimodal features are obtained by fusing the text features and the key image features. The multimodal features are encoded into a preset semantic space to obtain a multimodal feature encoding vector, thereby aligning the language information such as color, shape, size, etc. represented in the image with the text features of the same meaning, achieving better fusion encoding, and then subsequently improving the accuracy of the text structuring results when sequence labeling processing is performed based on the multimodal feature encoding vector and structured processing is performed on the text to be processed based on the target object label sequence.

[0150] In practical applications, taking the attribute label of a product as an example, when processing sequence annotation, for each position in the text, it is necessary to predict a set of "B, I" confidences for all attribute categories, as well as a "O" confidence, such as "B-battery power", "I-sleeve length", etc. However, the attribute ranges of different products are not exactly the same. For example, most clothing does not have a "battery power" attribute, and most digital products do not have a "sleeve length" attribute. If the result sequence prediction is performed for all products under the same maximum attribute range, there will be a large number of false positives in the prediction results, which will ultimately reduce the accuracy of structured data.

[0151] Based on this, the embodiment of the present invention may include the following steps when performing sequence labeling processing based on the multimodal feature encoding vector to obtain the target object label sequence: Figure 3a The following steps:

[0152] S301, determining a first target preset object among a plurality of preset objects based on the multimodal feature coding vector.

[0153] The first target preset object is a preset object existing in the text to be processed, that is, the first target preset object is a preset object corresponding to the current input data. It can be understood that at least one first target preset object can be determined.

[0154] In an exemplary embodiment, the above step S301 can be implemented as follows: Figure 3b As shown in, including:

[0155] S3011, extracting the feature coding vector at the location of the preset classification identifier from the multimodal feature coding vector to obtain a classification feature coding vector.

[0156] The preset classification identifier is an identifier added to the beginning of the text to be processed when generating the text feature. Exemplarily, the preset classification identifier is "[CLS]", and the classification feature encoding vector is the feature encoding of the corresponding position of the "[CLS]" symbol in the multimodal feature encoding vector.

[0157] S3013, performing classification processing based on the classification feature coding vector to obtain a classification result.

[0158] The classification result includes a confidence level corresponding to each preset object among a plurality of preset objects, and the confidence level represents the probability that the corresponding preset object exists in the text to be processed.

[0159] S3015: Determine a first target preset object among the multiple preset objects, the confidence level of which is greater than a confidence level threshold.

[0160] The confidence threshold can be set according to actual needs. The lower the confidence threshold, the larger the range of selected objects, which is more suitable for the situation where there are a large number of preset objects (such as attributes) in a single data. For example, the confidence threshold S thr The range can be set between 0.3 and 0.6.

[0161] In the above implementation, classification processing is performed within the range of multiple preset objects using the extracted classification feature coding vector to determine the probability of each preset object existing in the text to be processed, and then the first target preset object is quickly and accurately determined in combination with the confidence threshold.

[0162] S303: Determine the first target object prototype vector corresponding to the first target preset object in the object prototype vector set.

[0163] The object prototype vector set includes the object prototype vector of each preset object in the plurality of preset objects, and the object prototype vector corresponds to the preset object one by one. The prototype vector refers to a learnable vector parameter corresponding to a specific object category.

[0164] In a specific implementation, for a preset object whose confidence is greater than the confidence threshold, the corresponding position is set to 1 to represent selection, otherwise it is set to 0 to represent unselection, thereby obtaining a 0-1 array, which is used to select C in the object prototype vector pool P m object prototype vectors, and obtain the first target object prototype vector set P of the selected minimized range min , that is, the P min It can be expressed as the following formula (6):

[0165] P min =P[R i >S thr ] (6)

[0166] Among them, R i Indicates the probability that each preset object exists in the text to be processed, i.e., the confidence level; S thr Represents the confidence threshold.

[0167] S305: performing matching processing on the multimodal feature coding vector based on the first target object prototype vector to obtain a target multimodal feature coding vector corresponding to the first target preset object.

[0168] The matching process is used to determine the matching degree between the first target object prototype vector and each text position.

[0169] In a specific implementation, for each feature coding vector in the multimodal feature coding vector, the matching degree between the feature coding vector and the first target object prototype vector can be calculated, for example, cosine similarity can be calculated, and then the matching degree is used as a new channel to be spliced ​​with the corresponding feature coding vector. Assume that the feature coding vector at the corresponding text position j in the multimodal feature coding vector is c represents the feature encoding vector f j The matching degree with the first target object prototype vector, then the new feature (i.e. matching degree feature) c and the original feature (i.e. feature encoding vector) f j The channel ratio is 1:D.

[0170] By selecting C m The prototype vectors of each object are matched as above to obtain the corresponding C m C m target multimodal feature encoding vector.

[0171] S307: Perform sequence labeling processing based on the target multimodal feature encoding vector to obtain a target object label sequence corresponding to the first target preset object.

[0172] Specifically, for each target multimodal feature coding vector, a sequence labeling process is performed based on the target multimodal feature coding vector to obtain the "B, I, O" sequence result of the first target preset object corresponding to the target multimodal feature coding vector, that is, for each first target preset object at each position in the corresponding text, it is only necessary to predict the confidence corresponding to the three symbols "B", "I" and "O", and then for C m The first target preset object can get this C m The first target preset object corresponds to C m A “B, I, O” sequence result (i.e., the target object label sequence).

[0173] In getting this C m The first target preset object corresponds to C m After the "B, I, O" sequence results are obtained, for each first target preset object, the text to be processed is structured with its corresponding "B, I, O" sequence result, thereby obtaining corresponding structured data, which indicates the text information corresponding to the first target preset object in the text to be processed.

[0174] The above embodiment achieves the minimization of the object information scope guided by the object prototype. Compared with the related art, the object labels (such as B, I, O) of all preset objects must be predicted at each position of the text, and the number of classifications required to be performed is fixed to T. o=(C × |{B, I}| + |{O}|) × S. Where S is the sequence length of the text, and C is the number of preset objects, such as the types of attributes.

[0175] The above-mentioned embodiment adaptively determines the object range for the input text and image, selects the object prototype vector within the range, and only predicts the object label sequence within the selected object range, and no longer predicts the result sequence for the preset objects outside the range. Therefore, it only needs to predict the classification confidence of the preset objects existing in the text once. Since the number of preset objects with a confidence greater than the confidence threshold is C m << C, the number of classification times is T m = C + (|{B, I, O}| × S) × C m , at this time T m << T o , thus significantly reducing the false alarm situation in the result sequence and improving the accuracy of the structured data.

[0176] Taking the preset object as the attribute of the commodity as an example, as Figure 4 shown is an example of selecting the first target preset attribute from multiple preset attributes for the input text and the corresponding input image. The solid line box is the first target preset attribute selected when the input data is the text and image corresponding to shoes; the dashed line box is the first target preset attribute selected when the input data is the text and image corresponding to a dress. Subsequently, only the selected first target preset attribute will be predicted for the attribute label sequence, so as to obtain the target attribute label sequence corresponding to each first target preset attribute.

[0177] Considering that if the cosine similarity is directly used as a new channel to be concatenated with the features, the information proportion of the cosine similarity is relatively small, and the effect of object prototype guidance is poor. To improve the effect of object prototype guidance, in an exemplary embodiment, the above step S305 of performing matching processing on the multimodal feature encoding vector based on the first target object prototype vector to obtain the target multimodal feature encoding vector corresponding to the first target preset object may include:

[0178] Determine the matching degree between each feature encoding vector in the multimodal feature encoding vector and the first target object prototype vector;

[0179] Determine the product of the matching degree and the corresponding feature vector, and concatenate the result of the product with the corresponding feature vector in the channel dimension to obtain the target multimodal feature encoding vector.

[0180] Specifically, assume that the feature encoding vector corresponding to the text position j in the multimodal feature encoding vector is c represents this feature encoding vector f jThe matching degree with the first target object prototype vector is firstly calculated by comparing the matching degree (such as cosine similarity) c and f j Multiply them, and then add the product to f j In the channel dimension, if the matching feature is c·f j , then the new feature (i.e. matching feature) c·f j With the original feature f j The channel ratio of (i.e., feature encoding vector) is D:D.

[0181] Exemplarily, for the selected first target preset object a, each feature coding vector in the corresponding target multimodal feature coding vector can be calculated by the following formula (7):

[0182]

[0183] Among them, ||·|| is the modulus of the vector; represents the first target object prototype vector corresponding to the first target preset object a; f j Represents the feature encoding vector corresponding to the text position j in the multimodal feature encoding vector; Concate(·,·) is the concatenation function of the channel dimension.

[0184] By multiplying the matching degree with the corresponding feature coding vector and concatenating the result of the multiplication with the feature coding vector in the channel dimension, the proportion of matching degree information in the target multimodal feature coding vector can be increased, thereby enhancing the guiding effect of the object prototype vector, further reducing the false positives in the result sequence, and helping to improve the accuracy of structured data.

[0185] In a specific implementation, the data processing method of the embodiment of the present invention can be implemented based on a trained data processing model, that is, the data processing method of the embodiment of the present invention can also include a training step of the data processing model. Specifically, the training process of the data processing model can include the following steps:

[0186] Acquire sample data in a sample data set, wherein the sample data includes sample text and corresponding sample images, and the sample text corresponds to a reference object label sequence;

[0187] The sample data is input into a feature encoder of an initial data processing model to obtain an output sample multimodal feature encoding vector; the feature encoder is used to determine a sample text feature corresponding to the sample text based on a pre-trained word vector layer and a sample image feature corresponding to the sample image based on a pre-trained visual model, determine a sample key image feature in the sample image feature based on the sample text feature, obtain a sample multimodal feature by fusing the sample text feature and the sample key image feature, and obtain the sample multimodal feature encoding vector by encoding the sample multimodal feature based on a pre-trained language model;

[0188] Inputting the sample multimodal feature encoding vector into the feature decoder of the initial data processing model for sequence labeling processing to obtain prediction probabilities of multiple prediction object label sequences; the multiple prediction object label sequences include the reference object label sequence;

[0189] Determining a loss value based on the prediction probabilities of the multiple prediction object label sequences;

[0190] The model parameters in the initial data processing model are updated based on the loss value, and the iterative training is continued based on the updated model parameters until the training end condition is met, thereby obtaining the data processing model.

[0191] The sample text is the description text of the corresponding sample image. The training end condition may include that the number of iterations reaches a preset number of iterations, or the loss value reaches a preset loss threshold, or the difference between the two loss values ​​reaches a minimum difference threshold. The trained data processing model is the initial data processing model corresponding to the model parameters at the end of the training.

[0192] In the specific implementation, Figure 5a The structural diagram of the data processing model shown in the figure, wherein the feature encoder may include a pre-trained model and an attention network, and the pre-trained model includes a pre-trained word vector layer, a pre-trained visual model and a pre-trained language model; the feature decoder may adopt a basic BiLSTM+CRF neural network model, specifically including connecting BiLSTM, MLP and CRF in sequence.

[0193] The pre-trained word vector layer is used to determine the sample text features corresponding to the sample text. Specifically, each word and Chinese character in the sample text is tokenized through the dictionary preset by the pre-trained language model, converted into a fixed ID, and the corresponding word vector is found in the pre-trained word vector layer according to the ID, thereby converting the sample text into a sample word vector sequence. The pre-trained word vector layer can be trained together with the pre-trained language model during pre-training.

[0194] The pre-trained visual model is used to determine the sample image features corresponding to the sample image. The pre-trained visual model may include a multi-layer convolutional neural network. The sample image is processed by the multi-layer convolutional neural network of the pre-trained visual model and converted into a feature block with a specific spatial length and width (such as 7×7) and a specific number of channels (such as 1024). The feature block is mapped to the same number of channels as the word vector (such as 768) using MLP, and processed into a vector sequence of a specific length (such as 49) to obtain the sample image features. Exemplarily, the pre-trained visual model can be a ResNet, or a Vision Transformer type model, such as ViT (Vision Transformer).

[0195] The attention network is used to determine the sample key image features in the sample image features based on the sample text features. Specifically, the attention network may include a cross attention network, which uses the sample text features as a "query" input and the sample image features as a "key-value pair" input, and uses a cross attention mechanism to determine the sample key image features in the sample image features based on the sample text features. The sample key image features can be spliced ​​with the sample text features into a whole sequence feature, i.e., a sample multimodal feature, and then input into the pre-trained language model for encoding processing.

[0196] In an exemplary embodiment, Figure 5b As shown in the structural diagram of the data processing model, the attention network can also include a self-attention network, which takes sample text features as "query" input and sample image features as "key-value pair" input, and uses a self-attention mechanism to determine sample key text features in the sample text features based on the sample text features, thereby splicing the sample text features, the sample key text features and the sample key feature images to obtain the sample multimodal features, which are then input into the pre-trained language model for encoding processing.

[0197] In the specific implementation, in order to achieve splicing with sample text features, simple MLPs are used in the self-attention network and the cross-attention network to remap the key features obtained based on their respective attention mechanisms from the internal feature sequence dimension of the attention mechanism back to the text feature sequence dimension, and then added and fused with the sample text features into multimodal features.

[0198] Through the self-attention network and the cross-attention network, the same sample text features are used as the "query" input to extract useful information matching the same linguistic meaning in the sample text features and sample image features, thereby minimizing the range of key features in the sample text features and sample image features.

[0199] The pre-trained language model is used to encode the sample multimodal features to obtain the sample multimodal feature encoding vector, thereby jointly encoding language-related image information and text information into a unified semantic space. The pre-trained language model can be a BERT model, or other BERT-derived models, such as RoBERTa (A Robustly OptimizedBERT), ALBERT (A Lite BERT), etc.

[0200] BiLSTM includes a forward LSTM network layer and a backward LSTM network layer. The sequential sequence of the sample multimodal feature encoding vector is used as the input of the forward LSTM network layer, and its reverse sequence is used as the input of the backward LSTM network layer. At time t, the hidden state sequence output by the forward LSTM network layer and the hidden state sequence output by the backward LSTM network layer are spliced ​​by position to obtain a complete hidden state sequence.

[0201] Since it is necessary to predict two symbols, "B-attribute name, I-attribute name", for each preset object, plus the non-attribute "O", there are a total of C (i.e., the number of preset objects) classification confidences, so the above complete hidden state sequence needs to pass through a linear transformation layer MLP to map the hidden state vector of each position from m (representing the dimension of the hidden state vector) dimension to C dimension, and use the Sigmoid activation function to map it to the interval of 0.0~1.0. Therefore, for the sample multimodal feature encoding vector sequence of length N, there are C×N candidate paths (i.e., multiple predicted object label sequences). It can be understood that the C×N predicted object label sequences include the path annotated by the training data, that is, the reference object label sequence corresponding to the sample text.

[0202] In CRF, the sample text position i is predicted as a symbol y i The classification confidence of The symbol y in the transfer matrix T learned by CRF i Move to the next position of a symbol y i+1 The transition probability is recorded as the transition score The prediction probability S of a path (i.e., a preset object label sequence) can be calculated by the following formula (8):

[0203]

[0204] The above formula (8) can be used to calculate the prediction probabilities of all prediction object label sequences in the CRF, and then the loss value can be determined based on the prediction probabilities of all prediction object label sequences. Based on the loss value, the model parameters in the initial data processing model are updated using the back propagation optimization mechanism, and the iterative training is continued based on the updated model parameters until the training end conditions are met, thereby obtaining the data processing model.

[0205] In a specific implementation, the prediction probability hypothesis corresponding to the path labeled by the training data, that is, the reference object label sequence, can be selected from multiple prediction object label sequences and recorded as S RealPath , the training goal of the model is to make S RealPath Maximize, minimize the prediction probability of other paths, i.e., the remaining prediction object label sequences. The loss function can be the log-likelihood loss function, and the loss value can be calculated by the following formula (9):

[0206]

[0207] In the related art, during training, the parameters of the pre-trained model are usually fixed to extract feature information as the input of the downstream feature decoder. That is, when the model parameters are updated, only the model parameters in the feature decoder are updated and trained, thereby ignoring the differences between the upstream pre-training and the downstream feature decoding tasks, and reducing the accuracy of the trained data processing model.

[0208] Taking the multimodal attribute value extraction task for e-commerce advertising understanding as an example, from the perspective of language pre-training, the commonly used BERT model adopts a self-supervised learning approach. The data used is unlabeled corpus collected from websites such as Wikipedia, and self-supervised training is performed using tasks such as word completion and context judgment. From the perspective of visual pre-training, the commonly used ResNet model adopts a supervised learning approach. The data used is pictures from the ImageNet large-scale image database collected from the Internet and manually labeled with their categories, and supervised training is performed using image classification tasks. The input data used in the multimodal attribute value extraction task for e-commerce advertising understanding is text and pictures describing a product in an e-commerce scenario, which is quite different from the entry articles on Wikipedia or pictures of natural objects collected on the Internet.

[0209] Based on this, when the model parameters in the initial data processing model are updated based on the loss value, it can include: updating the model parameters in the feature encoder and the feature decoder respectively based on the loss value.

[0210] Specifically, while the model parameters in the feature decoder are updated based on the loss value, the model parameters in the feature encoder are also updated based on the loss value, that is, the parameters of multiple unimodal pre-trained models including the pre-trained word vector layer, the pre-trained language model and the pre-trained visual model are updated, so as to better utilize the upstream unimodal pre-training knowledge based on multi-model joint training and transfer it to the downstream multimodal tasks, thereby improving the training effect and thus improving the accuracy of the data processing model.

[0211] In a specific implementation, the model parameters in the feature encoder and the model parameters in the feature decoder can be loaded into the same deep learning computation training graph, and the model parameters on the deep learning training computation graph can be updated during training.

[0212] In an exemplary embodiment, when the model parameters in the feature encoder and the feature decoder are updated respectively based on the loss value, it may include:

[0213] Determine a first update amplitude of a first model parameter based on the loss value; the first model parameter is a model parameter in the pre-trained visual model;

[0214] Determine a second update amplitude of a second model parameter based on the loss value, the second update amplitude being greater than the first update amplitude; the second model parameter includes a model parameter in the feature encoder except the first model parameter and a model parameter in the feature decoder;

[0215] Corresponding model parameters are updated based on the first update amplitude and the second update amplitude respectively.

[0216] Specifically, considering the nature of downstream sequence annotation processing that is supplemented by vision and mainly based on text, for pre-trained visual models such as ResNet, a smaller learning rate, i.e., the first update amplitude, is set, for example, 5×10 -5 , and for the remaining model parameters of the feature encoder including the pre-trained language model such as BERT and the model parameters of the downstream feature decoder, a larger learning rate, i.e., the second update amplitude, is set, for example, 1×10 -4 , thereby further improving the training effect and the accuracy of the data processing model. In the specific implementation, the adjustment range of the learning rate hyperparameter can be set at 10 -5 ~10 -3 between.

[0217] In an exemplary embodiment, Figure 6 As shown, the sample multimodal feature encoding vector is input into the feature decoder of the initial data processing model for sequence labeling processing to obtain the prediction probability of multiple prediction object label sequences, which may include:

[0218] Inputting the sample multimodal feature encoding vector into the feature decoder to perform object range prediction to obtain an object range prediction result; the object range prediction result includes a second target preset object among a plurality of preset objects;

[0219] Determine the second target preset object and the corresponding second target object prototype vector in the object prototype vector set;

[0220] Performing matching processing on the sample multimodal feature coding vector based on the second target object prototype vector to obtain a target sample multimodal feature coding vector corresponding to the second target preset object;

[0221] The target sample multimodal feature encoding vector is input into the sequence labeling model of the feature decoder for sequence labeling to obtain the prediction probability of multiple prediction object label sequences corresponding to the second target preset object.

[0222] Exemplarily, inputting the sample multimodal feature encoding vector into the feature decoder to perform object range prediction, and obtaining the object range prediction result may include:

[0223] Extracting the feature coding vector at the location of the preset classification identifier from the sample multimodal feature coding vector to obtain the sample classification feature coding vector; the preset classification identifier is an identifier added to the beginning of the sample text when generating the sample text feature;

[0224] Inputting the sample classification feature encoding vector into the classification network of the feature decoder for classification processing to obtain a prediction classification result; the prediction classification result includes a prediction confidence corresponding to each preset object in a plurality of preset objects, and the prediction confidence represents the probability that the corresponding preset object exists in the sample text;

[0225] A second target preset object whose prediction confidence is greater than a confidence threshold is determined among the multiple preset objects to obtain an object range prediction result.

[0226] Specifically, the preset classification identifier is an identifier added to the beginning of the sample text when generating the sample text feature. Exemplarily, the preset classification identifier is "[CLS]", then the sample classification feature encoding vector is the feature encoding of the position corresponding to the "[CLS]" symbol in the sample multimodal feature encoding vector, and then the sample classification feature encoding vector is input to the classification network layer of the feature decoder, and the channel dimension of the sample classification feature encoding vector is mapped to the number of preset objects using linear mapping in the classification network layer, and mapped to the range of 0.0 to 1.0 using the Sigmoid activation function, so as to obtain the probability that each preset object exists in the sample text, that is, the prediction confidence. Specifically, the current input i (including the sample text T i and the corresponding sample image I i ) confidence level R i It can be expressed as the following formula (10):

[0227] R i =σ((f(T i , I i |θ)[″CLS″])W cls ) (10)

[0228] Among them, W cls represents the model parameters in the classification network; σ(·) represents the Sigmoid activation function; θ represents the model parameters in the feature encoder; f(·) represents the sample multimodal feature encoding vector.

[0229] The object prototype vector set includes an object prototype vector of each preset object in a plurality of preset objects, and the object prototype vectors correspond one-to-one to the preset objects. Each object prototype vector in the object prototype vector set is a vector parameter that can be learned together with the model during training.

[0230] For specific matching processing, please refer to the above Figure 3a The relevant contents in the illustrated method embodiment are not repeated here. In addition, after obtaining the target sample multimodal feature coding vector corresponding to the second target preset object, the target sample multimodal feature coding vector corresponding to the second target preset object is subjected to sequence labeling processing based on the sequence labeling model in the feature decoder. Specifically, the sequence labeling model may include a BiLSTM, an MLP and a CRF connected in sequence. The target sample multimodal feature coding vector corresponding to the second target preset object can be input into the sequence labeling model BiLSTM, and then the prediction probability of all the predicted object label sequences in the CRF can be calculated for the second target preset object through the sequential processing of the BiLSTM, the MLP and the CRF. The specific calculation method of the prediction probability can refer to the relevant contents of the aforementioned formulas (8) and (9), which will not be repeated here.

[0231] Since the probability that each preset object exists in the sample text is added in the object prototype guidance process, that is, the confidence R i Therefore, in an exemplary embodiment, the loss value determined based on the prediction probability of multiple prediction object label sequences may include:

[0232] Determining a first loss value based on the prediction probabilities of the plurality of prediction object label sequences corresponding to the second target preset object by means of a first loss function;

[0233] Determining a second loss value based on the predicted classification result and reference classification label information corresponding to the sample text by a second loss function; the reference classification label information indicates whether each of the plurality of preset objects exists in the sample text;

[0234] Based on the first loss value and the second loss value, a total loss value is determined, and then the model parameters of the initial data processing model can be updated based on the total loss value.

[0235] Specifically, the first loss function may be a log-likelihood loss function, and the second loss function may be a binary cross-entropy loss function. Then the total loss value L total It can be expressed as the following formula (11):

[0236] L total =L TAG Ten λL CLS (11)

[0237] Among them, L TAG represents the first loss value (calculated by referring to the above formula (9)), L CLS represents the second loss value, λ is the weighting coefficient. In order to ensure that the two loss functions contribute equally in the process of model parameter update, the weighting coefficient λ can be taken as 10 3 .

[0238] Among them, the second loss value L CLS It can be expressed as the following formula (12):

[0239]

[0240] Among them, C is the number of preset objects; y i Derived from the reference classification label information, it is the classification label of the i-th preset object. It is 1 if the preset object exists in the sample text, and 0 if it does not exist. i = 1, if the classification label of the preset object exists in the sample text, the corresponding category confidence p(y i ) is maximized; otherwise, the corresponding category confidence p(yi ) is minimized, that is, 1-p(y i ).

[0241] Assuming that the reference classification label information is {attribute a: 1, attribute b: 0, attribute c: 0, attribute d: 1}, it can be determined that attribute a and attribute d exist in the sample text, and attribute b and attribute c do not exist in the sample text.

[0242] Based on this, before training, it is also necessary to determine a plurality of preset objects (i.e., a preset object set) corresponding to the sample data set for the sample data set, that is, in an exemplary embodiment, the method may further include:

[0243] Determine the object corresponding to each sample text based on the reference object label sequence corresponding to each sample text in the sample data set;

[0244] Based on the object corresponding to each of the sample texts, generating a preset object set corresponding to the sample data set;

[0245] Based on the preset object set and the object corresponding to the sample text in each of the sample data, reference classification label information corresponding to the sample text is generated.

[0246] Taking the annotation format of the reference object label sequence as "B, I, O" as an example, since the object name will be annotated after the B and I characters, the object corresponding to each sample text can be determined based on the annotation sequence "B, I, O" corresponding to each sample text in the sample data set; then, after deduplication of the object corresponding to each sample text in the sample data set, a preset object set corresponding to the sample data set can be generated; then, for the sample text in each sample data, the classification label information is annotated based on the object corresponding to the sample text within the range of the preset object set to obtain the reference classification label information corresponding to the sample text in each sample data.

[0247] For example, assuming that the sample data set includes 3 sample data, based on the annotation sequence "B, I, O" of the sample text in each sample data, it can be determined that the attribute corresponding to sample text 1 in sample data 1 is {attribute a, attribute b}, the attribute corresponding to sample text 2 in sample data 2 is {attribute a, attribute c}, and the attribute corresponding to sample text 3 in sample data 3 is {attribute c}. After statistical deduplication of the attributes corresponding to the above sample texts, it can be obtained that the preset object set corresponding to the sample data set is {attribute a, attribute b, attribute c}. Then, the reference classification label information corresponding to sample text 1 in sample data 1 is {attribute a:1, attribute b:1, attribute c:0}, the reference classification label information corresponding to sample text 2 in sample data 2 is {attribute a:1, attribute b:0, attribute c:1}, and the reference classification label information corresponding to sample text 3 in sample data 3 is {attribute a:0, attribute b:0, attribute c:1}.

[0248] Considering that the second target preset object obtained based on the predicted classification result may be different from the reference classification label information, for example, given a sample data, the sample text has three attributes of "name", "style", and "size", and the corresponding "B, I, O" sequence annotation information, and the three attributes of "name", "style", and "type" in the predicted classification result output by the model in training through formula (10) are greater than the confidence threshold, and then the "B, I, O" sequence of the three attributes of "name", "style", and "type" is predicted through sequence annotation processing. Obviously, the prediction result is wrong (the "B, I, O" sequence of the attribute "type" should not be predicted, but the "B, I, O" sequence of the attribute "size" should be predicted).

[0249] Based on this, in an exemplary embodiment, in order to constrain the model to predict the correct sequence result, the above-mentioned first loss value determined based on the prediction probability of the multiple prediction object label sequences through the first loss function may include:

[0250] When the second target preset object matches the reference classification label information, taking the predicted probability corresponding to the reference object label sequence in the multiple predicted object label sequences as the maximization target of the first loss function, and determining a first loss value using the first loss function based on the maximization target and the predicted probability of the multiple predicted object label sequences;

[0251] When the second target preset object does not match the reference classification label information, the prediction probability corresponding to the target prediction object label sequence in the multiple prediction object label sequences is used as the maximization target of the first loss function, and the first loss value is determined using the first loss function based on the maximization target and the prediction probability of the multiple prediction object label sequences; each object label in the target prediction object label sequence represents a non-preset object.

[0252] Since the second target preset object is a preset object with a prediction confidence greater than the confidence threshold among multiple preset objects, that is, the selected second target preset object is likely to exist in the sample text. The reference classification label information can indicate which preset objects exist in the sample text. Therefore, if the preset objects existing in the sample text indicated by the reference classification label information include the second target preset object, it can be considered that the second target preset object matches the reference classification label information. Conversely, if the preset objects existing in the sample text indicated by the reference classification label information do not include the second target preset object, it can be considered that the second target preset object does not match the reference classification label information.

[0253] Among them, each object label in the target prediction object label sequence represents a non-preset object. Taking the "B, I, O" annotation format as an example, the target prediction object label sequence is a full "O" sequence.

[0254] Still using the above example to illustrate, for the "B, I, O" sequence corresponding to the correctly predicted attribute "name", the prediction probability corresponding to the reference attribute label sequence in the multiple predicted object label sequences corresponding to the attribute "name" is used as the maximization target of the first loss function, that is, S in the above formula (9). RealPath , and then the first loss value can be calculated based on the above formula (9); similarly, for the "B, I, O" sequence corresponding to the correctly predicted attribute "style", the prediction probability corresponding to the reference attribute label sequence in the multiple predicted object label sequences corresponding to the attribute "style" is used as the maximization target of the first loss function, that is, S in the above formula (9). RealPath , and then the first loss value can be calculated based on the above formula (9).

[0255] For the "B, I, O" sequence corresponding to the attribute "type" that is predicted incorrectly, the prediction probability corresponding to the entire "O" sequence (target attribute label sequence) is selected from the multiple predicted attribute sequences corresponding to this attribute as the maximization target of the first loss function, that is, S in the above formula (9): RealPath , and then the first loss value can be calculated based on the above formula (9).

[0256] For the missed prediction of the "size" attribute, although there is no corresponding prediction attribute label sequence and the first loss value cannot be constrained, the second loss value can be used to constrain the loss of the missed prediction of the attribute category.

[0257] By processing the above-mentioned first loss value, the model can learn to correctly handle the situation where the predicted classification results are erroneous by using the output target predicted object label sequence (such as the full "O" sequence in the B, I, O annotation format), so that the trained data processing model only predicts the object label sequence of the selected preset object.

[0258] In order to more clearly illustrate the technical solution of the embodiment of the present invention, Figure 6 , taking the text and images corresponding to e-commerce products, the attribute value extraction of product attributes and then obtaining the "attribute-attribute value" structured data is explained as an example.

[0259] like Figure 7 As shown, the text to be processed is "sports shoes with gradient colors...", and the corresponding image to be processed is an image of sports shoes. The text to be processed is input into the pre-trained word vector layer of the data processing model to obtain the output text feature sequence, and the image to be processed is input into the pre-trained visual model of the data processing model to obtain the output image feature sequence. The text feature sequence is processed based on the self-attention mechanism to obtain the key text feature sequence, and the text feature sequence (as the "query" Q input) and the image feature sequence (as the key value K, V input) are processed based on the cross-attention mechanism to obtain the key image feature sequence. The text feature sequence, the key text feature sequence and the key image feature sequence are spliced ​​to obtain a text-guided multimodal feature sequence. The text-guided multimodal feature sequence is input into the pre-trained language model for encoding to obtain a feature encoding vector sequence.

[0260] Extract the classification feature encoding vector corresponding to the position of the "[CLS]" symbol in the feature encoding vector sequence and input it into the classification network layer to obtain the confidence R of whether each preset attribute exists in the text i to be processed i , set the position corresponding to the preset attribute whose confidence is greater than the confidence threshold in the array composed of multiple preset attributes to 1, and set the position corresponding to the preset attribute whose confidence is less than or equal to the confidence threshold to 0, so as to obtain a 0-1 array such as 0,1,1,0...,1. Based on this 0-1 array, search the attribute prototype vector pool to obtain the corresponding target attribute prototype vector such as {P1,P2...,P n}.

[0261] For each target attribute prototype vector P i Perform matching processing, that is, calculate the prototype vector P of the target attribute iThe cosine similarity with each feature coding vector in the feature coding vector sequence is used as the attention weight to multiply the corresponding feature coding vector and then concatenated with the corresponding feature coding vector in the channel dimension to obtain the target feature coding vector sequence guided by each target attribute prototype vector.

[0262] For each target attribute prototype vector guided by the target feature encoding vector sequence, the target feature encoding vector sequence is input into BiLSTM, and after being processed by BiLSTM, MLP and CRF in sequence, the target "B, I, O" sequence of the target preset attribute corresponding to each target attribute prototype vector is output. Based on the target "B, I, O" sequence of each target preset attribute, the attribute value is extracted from the text to be processed, thereby obtaining the structured data of target preset attribute-attribute value.

[0263] Corresponding to the data processing methods provided in the above-mentioned embodiments, an embodiment of the present invention further provides a data processing device. Since the data processing device provided in the embodiment of the present invention corresponds to the data processing methods provided in the above-mentioned embodiments, the implementation methods of the aforementioned data processing methods are also applicable to the data processing device provided in this embodiment and will not be described in detail in this embodiment.

[0264] See also Figure 8 , which is a schematic diagram of the structure of a data processing device provided by an embodiment of the present invention, the device has the function of implementing the data processing method in the above method embodiment, and the function can be implemented by hardware or by hardware executing corresponding software. Figure 8 As shown, the data processing device 800 may include:

[0265] The data acquisition module 810 is used to acquire text features of the text to be processed and image features of the image to be processed; the text to be processed is a description text corresponding to the image to be processed;

[0266] A key image feature determination module 820 is used to determine a key image feature in the image features based on the text features; the key image feature is an image feature that has corresponding semantic information in the text features;

[0267] A fusion module 830 is used to obtain a multimodal feature by fusing the text feature and the key image feature, and encode the multimodal feature into a preset semantic space to obtain a multimodal feature encoding vector;

[0268] The sequence labeling module 840 is used to perform sequence labeling processing based on the multimodal feature encoding vector to obtain a target object label sequence; the target object label sequence is used to perform structured processing on the text to be processed to obtain target structured data.

[0269] In an exemplary embodiment, the key image feature determination module 820 includes:

[0270] A cross attention weight determination module, used for determining a cross attention weight based on the text feature and the image feature;

[0271] The key image feature determination submodule is used to adjust the image features based on the cross attention weights to obtain key image features.

[0272] In an exemplary embodiment, the fusion module 830 includes:

[0273] A self-attention weight determination module, used to determine the self-attention weight based on the text feature;

[0274] A key text feature determination module, used for adjusting the text feature based on the self-attention weight to obtain a key text feature;

[0275] The fusion submodule is used to fuse the text features, the key image features and the key text features to obtain multimodal features.

[0276] In an exemplary embodiment, the sequence tagging module 840 includes:

[0277] A first target preset object determination module, used to determine a first target preset object among a plurality of preset objects based on the multimodal feature encoding vector; the first target preset object is a preset object existing in the text to be processed;

[0278] A first determination module is used to determine a first target object prototype vector corresponding to the first target preset object in an object prototype vector set; the object prototype vector set includes an object prototype vector of each preset object in the plurality of preset objects;

[0279] A first matching processing module, configured to perform matching processing on the multimodal feature coding vector based on the first target object prototype vector to obtain a target multimodal feature coding vector corresponding to the first target preset object;

[0280] The sequence labeling submodule is used to perform sequence labeling processing based on the target multimodal feature encoding vector to obtain a target object label sequence corresponding to the first target preset object.

[0281] In an exemplary embodiment, the first target preset object determination module includes:

[0282] A classification feature coding determination module is used to extract a feature coding vector at a location of a preset classification identifier from the multimodal feature coding vector to obtain a classification feature coding vector; the preset classification identifier is an identifier added to the beginning of the text to be processed when generating the text feature;

[0283] A first classification processing module is used to perform classification processing based on the classification feature coding vector to obtain a classification result; the classification result includes a confidence level corresponding to each preset object in a plurality of preset objects, and the confidence level represents the probability that the corresponding preset object exists in the text to be processed;

[0284] The second determination module is used to determine a first target preset object among the multiple preset objects whose confidence level is greater than a confidence threshold.

[0285] In an exemplary embodiment, the first matching processing module includes:

[0286] A matching degree determination module, used to determine the matching degree between each feature coding vector in the multimodal feature coding vector and the first target object prototype vector;

[0287] The splicing processing module is used to determine the product of the matching degree and the corresponding feature vector, and splice the result of the product with the corresponding feature vector in the channel dimension to obtain a target multimodal feature coding vector.

[0288] In an exemplary embodiment, the apparatus further comprises a training module, wherein the training module comprises:

[0289] A sample data acquisition module is used to acquire sample data in a sample data set, wherein the sample data includes sample text and corresponding sample images, and the sample text corresponds to a reference object label sequence;

[0290] A sample multimodal feature encoding module, used for inputting the sample data into a feature encoder of an initial data processing model to obtain an output sample multimodal feature encoding vector; the feature encoder is used for determining a sample text feature corresponding to the sample text based on a pre-trained word vector layer and a sample image feature corresponding to the sample image based on a pre-trained visual model, determining a sample key image feature in the sample image feature based on the sample text feature, performing a fusion process based on the sample text feature and the sample key image feature to obtain a sample multimodal feature, and performing an encoding process on the sample multimodal feature based on a pre-trained language model to obtain the sample multimodal feature encoding vector;

[0291] A sequence probability prediction module, used for inputting the sample multimodal feature encoding vector into the feature decoder of the initial data processing model for sequence labeling processing to obtain prediction probabilities of multiple prediction object label sequences; the multiple prediction object label sequences include the reference object label sequence;

[0292] A loss value determination module, used to determine a loss value based on the prediction probabilities of the plurality of prediction object label sequences;

[0293] A parameter updating training module is used to update the model parameters in the initial data processing model based on the loss value until the training end condition is met to obtain the data processing model.

[0294] In an exemplary embodiment, the parameter update training module is specifically configured to update the model parameters in the feature encoder and the feature decoder respectively based on the loss value.

[0295] In an exemplary embodiment, the parameter updating training module includes:

[0296] A first update amplitude determination module, configured to determine a first update amplitude of a first model parameter based on the loss value; the first model parameter is a model parameter in the pre-trained visual model;

[0297] A second update amplitude determination module is used to determine a second update amplitude of a second model parameter based on the loss value, wherein the second update amplitude is greater than the first update amplitude; the second model parameter includes a model parameter in the feature encoder except the first model parameter and a model parameter in the feature decoder;

[0298] An updating submodule is used to update corresponding model parameters based on the first updating amplitude and the second updating amplitude respectively.

[0299] In an exemplary embodiment, the sequence probability prediction module includes:

[0300] An object range prediction module, used for inputting the sample multimodal feature encoding vector into the feature decoder to perform object range prediction, and obtain an object range prediction result; the object range prediction result includes a second target preset object among a plurality of preset objects;

[0301] A third determining module is used to determine a second target object prototype vector corresponding to the second target preset object in the object prototype vector set;

[0302] A second matching processing module, used for performing matching processing on the sample multimodal feature coding vector based on the second target object prototype vector to obtain a target sample multimodal feature coding vector corresponding to the second target preset object;

[0303] The sequence probability prediction submodule is used to input the target sample multimodal feature encoding vector into the sequence labeling model of the feature decoder for sequence labeling, so as to obtain the prediction probability of multiple predicted object label sequences corresponding to the second target preset object.

[0304] In an exemplary embodiment, the object range prediction module includes:

[0305] A sample classification feature coding extraction module is used to extract a feature coding vector at a location of a preset classification identifier from the sample multimodal feature coding vector to obtain a sample classification feature coding vector; the preset classification identifier is an identifier added to the beginning of the sample text when generating the sample text feature;

[0306] A second classification processing module is used to input the sample classification feature encoding vector into the classification network of the feature decoder for classification processing to obtain a prediction classification result; the prediction classification result includes a prediction confidence corresponding to each preset object in a plurality of preset objects, and the prediction confidence represents the probability that the corresponding preset object exists in the sample text;

[0307] The fourth determination module is used to determine a second target preset object among the multiple preset objects, the prediction confidence of which is greater than a confidence threshold.

[0308] In an exemplary embodiment, the loss value determination module includes:

[0309] A first loss value determination module, configured to determine a first loss value based on the prediction probabilities of the plurality of prediction object label sequences corresponding to the second target preset object through a first loss function;

[0310] A second loss value determination module, configured to determine a second loss value based on the predicted classification result and reference classification label information corresponding to the sample text by using a second loss function; the reference classification label information indicates whether each of the plurality of preset objects exists in the sample text;

[0311] A total loss value determination module is used to determine a total loss value based on the first loss value and the second loss value.

[0312] In an exemplary embodiment, the first loss value determination module includes:

[0313] A first determination submodule is configured to, when the second target preset object matches the reference classification label information, use the predicted probability corresponding to the reference object label sequence in the multiple predicted object label sequences as the maximization target of the first loss function, and determine a first loss value using the first loss function based on the maximization target and the predicted probability of the multiple predicted object label sequences;

[0314] The second determination submodule is used to use the prediction probability corresponding to the target prediction object label sequence in the multiple prediction object label sequences as the maximization target of the first loss function when the second target preset object does not match the reference classification label information, and determine the first loss value using the first loss function based on the maximization target and the prediction probability of the multiple prediction object label sequences; each object label in the target prediction object label sequence represents a non-preset object.

[0315] In an exemplary embodiment, the training module further includes:

[0316] A fifth determination module, configured to determine the object corresponding to each sample text based on a reference object label sequence corresponding to each sample text in the sample data set;

[0317] A preset object set determination module, used to generate a preset object set corresponding to the sample data set based on the object corresponding to each of the sample texts;

[0318] The reference classification label information generating module is used to generate reference classification label information corresponding to the sample text based on the preset object set and the object corresponding to the sample text in each sample data.

[0319] It should be noted that the device provided in the above embodiment, when implementing its functions, is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0320] An embodiment of the present invention provides an electronic device, which includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement any one of the data processing methods provided in the above method embodiments.

[0321] The memory can be used to store software programs and modules. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, application programs required for functions, etc.; the data storage area can store data created according to the use of the device, etc. In addition, the memory can include a high-speed random access memory and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory can also include a memory controller to provide the processor with access to the memory.

[0322] The method embodiments provided in the embodiments of the present invention may be executed in a computer terminal, a server or a similar computing device, that is, the above-mentioned electronic device may include a computer terminal, a server or a similar computing device. Fig. 9 is a hardware structure block diagram of an electronic device running a data processing method provided by an embodiment of the present invention, such as Fig. 9 As shown, the internal structure of the electronic device may include but is not limited to: a processor, a network interface and a memory. The processor, network interface and memory in the electronic device may be connected via a bus or other means. Fig. 9 The example of connecting through bus is taken in the following.

[0323] Among them, the processor (or CPU (Central Processing Unit)) is the computing core and control core of the electronic device. The network interface may optionally include a standard wired interface, a wireless interface (such as WI-FI, a mobile communication interface, etc.). Memory is a memory device in an electronic device for storing programs and data. It is understandable that the memory here can be a high-speed RAM storage device, or a non-volatile storage device (non-volatile memory), such as at least one disk storage device; optionally, it can also be at least one storage device located away from the aforementioned processor. The memory provides a storage space, which stores the operating system of the electronic device, which may include but is not limited to: Windows system (an operating system), Linux (an operating system), Android (Android, a mobile operating system) system, IOS (a mobile operating system) system, etc., and the present invention is not limited to this; and, in the storage space, one or more instructions suitable for being loaded and executed by the processor are also stored, and these instructions may be one or more computer programs (including program codes). In the embodiment of this specification, the processor loads and executes one or more instructions stored in the memory to implement the data processing method provided in the above method embodiment.

[0324] An embodiment of the present invention also provides a computer-readable storage medium, which can be set in an electronic device to store at least one instruction or at least one program related to implementing a data processing method. The at least one instruction or the at least one program is loaded and executed by the processor to implement any one of the data processing methods provided in the above method embodiments.

[0325] The embodiments of the present invention further provide a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of an electronic device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the electronic device performs any of the above-mentioned data processing methods.

[0326] Optionally, in this embodiment, the above-mentioned storage medium may include but is not limited to: a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and other media that can store program codes.

[0327] It should be noted that the sequence of the embodiments of the present invention described above is for description only and does not represent the advantages and disadvantages of the embodiments. The above describes specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0328] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0329] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0330] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A data processing method, characterized in that: The method comprises: Acquire text features of a text to be processed and image features of an image to be processed; the text to be processed is a description text corresponding to the image to be processed; Determine a key image feature in the image feature based on the text feature; the key image feature is an image feature having corresponding semantic information in the text feature; Based on the text feature and the key image feature, a fusion process is performed to obtain a multimodal feature, and the multimodal feature is encoded into a preset semantic space to obtain a multimodal feature encoding vector; Based on the multimodal feature encoding vector, determining a first target preset object among a plurality of preset objects; the first target preset object is a preset object existing in the text to be processed; Determine a first target object prototype vector corresponding to the first target preset object in an object prototype vector set; the object prototype vector set includes an object prototype vector of each preset object in the plurality of preset objects; Performing matching processing on the multimodal feature coding vector based on the first target object prototype vector to obtain a target multimodal feature coding vector corresponding to the first target preset object; Sequence labeling processing is performed based on the target multimodal feature encoding vector to obtain a target object label sequence corresponding to the first target preset object; the target object label sequence is used to perform structured processing on the text to be processed to obtain target structured data.

2. The method according to claim 1, characterized in that: The determining the key image features in the image features based on the text features includes: Determining a cross attention weight based on the text feature and the image feature; The image features are adjusted based on the cross attention weights to obtain key image features.

3. The method according to claim 2, characterized in that The fusing process based on the text feature and the key image feature to obtain the multimodal feature includes: Determining a self-attention weight based on the text feature; Adjusting the text feature based on the self-attention weight to obtain a key text feature; The text features, the key image features and the key text features are fused to obtain multimodal features.

4. The method according to claim 1, characterized in that: The determining, based on the multimodal feature encoding vector, a first target preset object among a plurality of preset objects comprises: Extracting a feature coding vector at a location of a preset classification identifier from the multimodal feature coding vector to obtain a classification feature coding vector; the preset classification identifier is an identifier added to the beginning of the text to be processed when generating the text feature; Performing classification processing based on the classification feature coding vector to obtain a classification result; the classification result includes a confidence level corresponding to each preset object in a plurality of preset objects, and the confidence level represents the probability that the corresponding preset object exists in the text to be processed; A first target preset object, among the plurality of preset objects, whose confidence level is greater than a confidence level threshold, is determined.

5. The method according to claim 1, characterized in that The matching process of the multimodal feature coding vector based on the first target object prototype vector to obtain the target multimodal feature coding vector corresponding to the first target preset object includes: Determining a degree of match between each feature encoding vector in the multimodal feature encoding vector and the first target object prototype vector; The product of the matching degree and the corresponding feature vector is determined, and the result of the product is concatenated with the corresponding feature vector in the channel dimension to obtain a target multimodal feature encoding vector.

6. The method according to any one of claims 1 to 5, characterized in that: The method is implemented based on a data processing model, and the method further includes training the data processing model, wherein the training of the data processing model includes: Acquire sample data in a sample data set, wherein the sample data includes sample text and corresponding sample images, and the sample text corresponds to a reference object label sequence; The sample data is input into a feature encoder of an initial data processing model to obtain an output sample multimodal feature encoding vector; the feature encoder is used to determine a sample text feature corresponding to the sample text based on a pre-trained word vector layer and a sample image feature corresponding to the sample image based on a pre-trained visual model, determine a sample key image feature in the sample image feature based on the sample text feature, obtain a sample multimodal feature by fusing the sample text feature and the sample key image feature, and obtain the sample multimodal feature encoding vector by encoding the sample multimodal feature based on a pre-trained language model; Inputting the sample multimodal feature encoding vector into the feature decoder of the initial data processing model for sequence labeling processing to obtain prediction probabilities of multiple prediction object label sequences; the multiple prediction object label sequences include the reference object label sequence; Determining a loss value based on the prediction probabilities of the multiple prediction object label sequences; Based on the loss value, the model parameters in the feature encoder and the feature decoder are updated respectively until the training end condition is met to obtain the data processing model.

7. The method according to claim 6, characterized in that The updating of the model parameters in the feature encoder and the feature decoder respectively based on the loss value comprises: Determine a first update amplitude of a first model parameter based on the loss value; the first model parameter is a model parameter in the pre-trained visual model; Determine a second update amplitude of a second model parameter based on the loss value, the second update amplitude being greater than the first update amplitude; the second model parameter includes a model parameter in the feature encoder except the first model parameter and a model parameter in the feature decoder; Corresponding model parameters are updated based on the first update amplitude and the second update amplitude respectively.

8. The method according to claim 6, characterized in that The step of inputting the sample multimodal feature encoding vector into the feature decoder of the initial data processing model for sequence labeling to obtain the prediction probabilities of multiple prediction object label sequences includes: Inputting the sample multimodal feature encoding vector into the feature decoder to perform object range prediction to obtain an object range prediction result; the object range prediction result includes a second target preset object among a plurality of preset objects; Determine the second target preset object and the corresponding second target object prototype vector in the object prototype vector set; Performing matching processing on the sample multimodal feature coding vector based on the second target object prototype vector to obtain a target sample multimodal feature coding vector corresponding to the second target preset object; The target sample multimodal feature encoding vector is input into the sequence labeling model of the feature decoder for sequence labeling to obtain the prediction probability of multiple predicted object label sequences corresponding to the second target preset object.

9. The method according to claim 8, characterized in that The step of inputting the sample multimodal feature encoding vector into the feature decoder to perform object range prediction and obtain the object range prediction result comprises: Extracting the feature coding vector at the location of the preset classification identifier from the sample multimodal feature coding vector to obtain the sample classification feature coding vector; the preset classification identifier is an identifier added to the beginning of the sample text when generating the sample text feature; Inputting the sample classification feature encoding vector into the classification network of the feature decoder for classification processing to obtain a prediction classification result; the prediction classification result includes a prediction confidence corresponding to each preset object in a plurality of preset objects, and the prediction confidence represents the probability that the corresponding preset object exists in the sample text; A second target preset object whose prediction confidence is greater than a confidence threshold is determined among the multiple preset objects to obtain an object range prediction result.

10. The method according to claim 9, characterized in that The determining of the loss value based on the prediction probabilities of the multiple prediction object label sequences includes: Determining a first loss value based on the prediction probabilities of the plurality of prediction object label sequences corresponding to the second target preset object by means of a first loss function; Determining a second loss value based on the predicted classification result and reference classification label information corresponding to the sample text by a second loss function; the reference classification label information indicates whether each of the plurality of preset objects exists in the sample text; Based on the first loss value and the second loss value, a total loss value is determined.

11. The method according to claim 10, characterized in that The first loss value is determined based on the prediction probabilities of the plurality of prediction object label sequences corresponding to the second target preset object by using the first loss function, including: When the second target preset object matches the reference classification label information, taking the predicted probability corresponding to the reference object label sequence in the multiple predicted object label sequences as the maximization target of the first loss function, and determining a first loss value using the first loss function based on the maximization target and the predicted probability of the multiple predicted object label sequences; When the second target preset object does not match the reference classification label information, the prediction probability corresponding to the target prediction object label sequence in the multiple prediction object label sequences is used as the maximization target of the first loss function, and the first loss value is determined using the first loss function based on the maximization target and the prediction probability of the multiple prediction object label sequences; each object label in the target prediction object label sequence represents a non-preset object.

12. A data processing device, characterized in that: The device comprises: A data acquisition module, used to acquire text features of a text to be processed and image features of an image to be processed; the text to be processed is a description text corresponding to the image to be processed; A key image feature determination module, used to determine a key image feature in the image features based on the text features; the key image feature is an image feature having corresponding semantic information in the text features; A fusion module, configured to obtain a multimodal feature by fusing the text feature and the key image feature, and encode the multimodal feature into a preset semantic space to obtain a multimodal feature encoding vector; A sequence labeling module is used to determine a first target preset object among multiple preset objects based on the multimodal feature coding vector; the first target preset object is a preset object existing in the text to be processed; determine the first target object prototype vector corresponding to the first target preset object in the object prototype vector set; the object prototype vector set includes the object prototype vector of each preset object in the multiple preset objects; match the multimodal feature coding vector based on the first target object prototype vector to obtain a target multimodal feature coding vector corresponding to the first target preset object; perform sequence labeling based on the target multimodal feature coding vector to obtain a target object label sequence corresponding to the first target preset object; the target object label sequence is used to perform structured processing on the text to be processed to obtain target structured data.

13. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by the processor to implement the data processing method according to any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by a processor to implement the data processing method according to any one of claims 1 to 11.

Images