Data distribution method and model training method, device, computer equipment and medium

Abstract

The application relates to the technical field of artificial intelligence, in particular to a data distribution method and device, a model training method and device, computer equipment and a medium. The method occludes text data, trains a preliminarily trained text encoder according to the text data and the occluded data, splices the text data and first distribution information, inputs the preliminarily trained text encoder, obtains a text feature vector, splices image data and second distribution information, inputs an image encoder, obtains an image feature vector, calculates the inner product of each text feature vector and each image feature vector, re-trains the text encoder according to the inner product, obtains a finally trained text encoder, preliminarily trains the text encoder by using the text data, so that the text encoder can learn information of a scene to which the text data belongs, and the text encoder is re-trained in combination with the distribution information and the image feature vector, the feature extraction capability of the model is improved, and the accuracy of data distribution is improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a data distribution method, a model training method, an apparatus, a computer device, and a medium. Background Technology

[0002] Currently, with the rapid development of artificial intelligence, intelligent applications have been widely deployed in various scenarios. For example, in the data distribution scenario, the distribution task can refer to distributing different data to different processing personnel. Existing methods usually use classification models or ranking and recall models to process the received text data and obtain the distribution information of the corresponding text data, which can effectively reduce labor costs and improve the efficiency of data distribution.

[0003] However, the received data is often represented in the form of image and text data. Existing methods ignore the image data associated with the text data and only process the text data. This results in the loss of additional scene information that the image data can provide, as well as missing information in the text data. Consequently, the data distribution accuracy is low. Therefore, how to effectively improve the data distribution accuracy has become an urgent problem to be solved. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a data distribution method, a model training method, an apparatus, a computer device, and a medium to solve the problem of low distribution accuracy in data distribution.

[0005] In a first aspect, embodiments of the present invention provide a model training method for data distribution, the model training method comprising: Get The dataset consists of groups of image and text data to be processed. Each group includes one image and one text data. Integers greater than zero; Will Each text data point is occluded separately to obtain... Occlusion data corresponding to text data, according to the The text data and the Using occlusion data corresponding to text data, a pre-set text encoder is trained to obtain a pre-trained text encoder. The Each text data point is concatenated with its corresponding first distribution information, and then input into the pre-trained text encoder for feature extraction. Each corresponding text feature vector; Will Each image data point is concatenated with its corresponding second segmentation information, and then input into a preset image encoder for feature extraction, resulting in... Image feature vectors corresponding to each image data; Calculate the sum of each text feature vector. The inner product between each image feature vector is used to calculate the feature extraction loss based on all the calculated inner products and the preset loss function. The pre-trained text encoder is then retrained based on the feature extraction loss to obtain the final trained text encoder. The final trained text encoder and the image encoder are used to form a trained data distribution model.

[0006] Secondly, embodiments of the present invention provide a data distribution method, the data distribution method comprising: Acquire the data to be distributed. When the data to be distributed belongs to image and text data, input the data to be distributed into the trained data distribution model for feature extraction to obtain image and text feature vectors. The image and text feature vectors are retrieved from a preset database to obtain at least one image and text storage vector. The image and text feature vectors are concatenated with each image and text storage vector, and the concatenation results are input into a preset fully connected layer for similarity calculation to obtain the similarity score of the corresponding image and text storage vector. The image and text storage vector corresponding to the maximum value among all similarity scores is determined as the reference vector, and the distribution information corresponding to the reference vector is used as the distribution information of the data to be distributed.

[0007] The trained data distribution model is obtained based on the data distribution model training method according to any one of claims 1-6.

[0008] Thirdly, embodiments of the present invention provide a model training apparatus for data distribution, the model training apparatus comprising: The data acquisition module is used to acquire... The dataset consists of groups of image and text data to be processed. Each group includes one image and one text data. Integers greater than zero; Text masking module, used to... Each text data point is occluded separately to obtain... Occlusion data corresponding to text data, according to the The text data and the Using occlusion data corresponding to text data, a pre-set text encoder is trained to obtain a pre-trained text encoder. The first feature extraction module is used to extract the features of the first feature extraction module. Each text data point is concatenated with its corresponding first distribution information, and then input into the pre-trained text encoder for feature extraction. Each corresponding text feature vector; The second feature extraction module is used to extract features from the second feature extraction module. Each image data point is concatenated with its corresponding second segmentation information, and then input into a preset image encoder for feature extraction, resulting in... Image feature vectors corresponding to each image data; The model training module is used to calculate the sum of each text feature vector and... The inner product between each image feature vector is used to calculate the feature extraction loss based on all the calculated inner products and the preset loss function. The pre-trained text encoder is then retrained based on the feature extraction loss to obtain the final trained text encoder. The model determination module is used to determine the final trained text encoder and the image encoder to form a trained data distribution model.

[0009] Fourthly, embodiments of the present invention provide a computer device, the computer device including a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the model training method as described in the first aspect.

[0010] Fifthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the model training method as described in the first aspect.

[0011] The beneficial effects of the embodiments of the present invention compared with the prior art are as follows: Get The data consists of a set of image and text data to be processed. Each set includes one image and one text data. Each text data point is occluded separately to obtain... Occlusion data corresponding to text data, based on Text data and Using occlusion data corresponding to the text data, a pre-defined text encoder is trained to obtain a pre-trained text encoder. Each text data point is concatenated with its corresponding first distribution information, and then input into a pre-trained text encoder for feature extraction. Each corresponding text feature vector of the text data will Each image data point is concatenated with its corresponding second segmentation information, and then input into a preset image encoder for feature extraction, resulting in... For each corresponding image data point, calculate the image feature vector and the text feature vector respectively. The inner product between each image feature vector is used to calculate the feature extraction loss based on all the calculated inner products and a preset loss function. The pre-trained text encoder is then retrained based on this feature extraction loss to obtain the final trained text encoder. The final trained text encoder and the image encoder form a trained data distribution model. The text encoder is pre-trained using scene-appropriate text data, enabling it to learn scene information. The text encoder is then re-trained using the distribution information and text data, and image features are introduced as a reference. This allows the text encoder to align text feature vectors and image feature vectors belonging to the same distribution information while learning the distribution information, improving the expressive power of the features extracted by the data distribution model and thus increasing the accuracy of data distribution. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a schematic diagram of an application environment for a data distribution model training method provided in Embodiment 1 of the present invention; Figure 2 This is a schematic flowchart of a data distribution model training method provided in Embodiment 1 of the present invention; Figure 3 This is a flowchart illustrating a data distribution method provided in Embodiment 2 of the present invention; Figure 4 This is a schematic diagram of a data distribution model training device provided in Embodiment 3 of the present invention; Figure 5 This is a schematic diagram of the structure of a computer device provided in Embodiment 4 of the present invention. Detailed Implementation

[0014] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of the invention. However, those skilled in the art will understand that the invention can be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of the invention with unnecessary detail.

[0015] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0016] It should also be understood that the term “and / or” as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0017] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0018] Furthermore, in the description of this invention and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0019] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of the invention include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0020] The embodiments of this invention can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0021] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0022] It should be understood that the sequence number of each step in the following embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0023] To illustrate the technical solution of the present invention, specific embodiments are described below.

[0024] The data distribution model training method provided in Embodiment 1 of this invention can be applied to, for example... Figure 1 In this application environment, the client communicates with the server. The client includes, but is not limited to, PDAs, desktop computers, laptops, ultra-mobile personal computers (UMPCs), netbooks, cloud terminal devices, and personal digital assistants (PDAs). The server can be a standalone server or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0025] The client and server can be deployed in a task distribution scenario, such as a government task distribution scenario, a police task distribution scenario, or a sanitation task distribution scenario. The client can be used to process the data corresponding to the task and determine the processing unit to which the data corresponding to the task should be sent, that is, to determine the distribution information of the data corresponding to the task. Therefore, the client can communicate with at least one distribution processing terminal to transmit data distribution information. The server can be used to store the received task data and can communicate with at least one task receiving terminal to obtain task data from different task receiving terminals.

[0026] For example, in a government affairs dispatching scenario, the server can communicate with at least one 12345 government service hotline receiving terminal. The 12345 government service hotline receiving terminal can refer to a hotline receiving telephone used to answer calls from the public to obtain their requests. The receiving terminal needs to be configured with operators. After the operators communicate with the public by phone and obtain the affairs data, they send the affairs data to the server for storage, waiting for the client to dispatch the data for processing. The data is then dispatched to the appropriate processing unit. For example, if the affairs data involves illegal street vending, after the client processes it, the affairs data will be dispatched to the urban management unit for processing. This can reduce the resource cost of manpower for affairs dispatching and improve the efficiency of affairs data dispatching.

[0027] See Figure 2 This is a flowchart illustrating a data distribution model training method provided in Embodiment 1 of the present invention. The above-described data distribution model training method can be applied to... Figure 1 The client, corresponding to a computer device, connects to the server to obtain text and image data to be processed. "To be processed" may refer to data requiring further processing. The client's computer device is equipped with a pre-defined text encoder and a pre-defined image encoder. The pre-defined text encoder can be a randomly initialized text encoder, and the pre-defined image encoder can be a pre-trained image encoder. For example... Figure 2 As shown, the model training method for this data distribution may include the following steps: Step S201, obtain The data consists of a set of image and text data to be processed, each set including one image and one text data.

[0028] in, The value is a positive integer. "To be processed" can refer to items that need to be allocated. A single set of image and text data corresponds to an item to be allocated. A single set of image and text data contains an image data and a text data that have a corresponding relationship. That is, the image data and the text data both correspond to the item to be allocated. Items to be allocated can refer to government affairs, police affairs, sanitation affairs, etc.

[0029] Specifically, The group of text and image data to be processed can refer to the data within a preset historical time period in the data distribution task. The graphic and textual data corresponding to each item to be allocated can be set by the implementer. The preset historical time period can be one week or one month before the current time, or one week or one month from any historical time.

[0030] It should be noted that since the image and text data were collected within a historical time period, the default... Each group of text and image data to be processed has a corresponding distribution processing result, i.e., distribution information.

[0031] The above acquisition The steps to obtain grouped image and text data, each group consisting of one image and one text data, are as follows: The data to be processed is used as training samples for subsequent model training, thereby improving the data distribution model's ability to extract features from the data to be distributed, and thus improving the data distribution accuracy of the data distribution model.

[0032] Step S202, will Each text data point is occluded separately to obtain... Occlusion data corresponding to text data, based on Text data and Using occlusion data corresponding to the text data, a pre-set text encoder is trained to obtain a pre-trained text encoder.

[0033] Occlusion processing can refer to randomly occluding the words contained in the text data. Occlusion data can refer to text data containing occluded words. A preliminarily trained text encoder can refer to a text encoder that has a preliminary ability to extract text data features. A preliminarily trained text encoder can be an encoder such as the Roberta model encoder or the Transformer model encoder.

[0034] Specifically, Each piece of text data needs to undergo size normalization to ensure it fits the input size of the text encoder. For any given piece of text data, let it contain... Each term is assigned the same sampling probability. Sampling is performed according to the sampling probability distribution of all terms. Since the sampling probability of each term is the same, it can be considered as random sampling. The randomly sampled terms are masked, and masking can be a zeroing operation.

[0035] During the occlusion operation, the same text data can be sampled multiple times to obtain multiple occlusion data corresponding to the text data. Then, based on... A pre-set text encoder is trained using a set of text data and at least one corresponding occlusion data to obtain a pre-trained text encoder.

[0036] In one embodiment, the same text data can be sampled multiple times to obtain a single occlusion data containing multiple occluded terms.

[0037] Optionally, according to Text data and Using occlusion data corresponding to the text data, a pre-defined text encoder is trained to obtain a pre-trained text encoder, including: For any occlusion data in corresponding text data, the occlusion data is input into a preset text encoder for feature extraction to obtain occlusion features; The occlusion features are input into a preset text encoder for feature reconstruction to obtain reconstructed data; Based on the text data corresponding to the reconstructed data and occlusion data, and the preset reconstruction loss function, calculate the reconstruction sub-loss; By iterating through the occlusion data of each corresponding text data, the reconstruction sub-loss of the corresponding text data is obtained. The reconstruction sub-losses of all corresponding text data are added together to obtain the reconstruction loss. Based on the reconstruction loss, the preset text encoder is initially trained to obtain the pre-trained text encoder.

[0038] Among them, occlusion features can be used to characterize the feature information of occluded data, reconstructed data can refer to the unoccluded data predicted by the preset text encoder, the preset reconstruction loss function can be the mean squared error loss function, the reconstruction sub-loss can refer to the reconstruction loss of the occluded data, and the reconstruction loss can refer to the reconstruction loss of all occluded data.

[0039] Specifically, the preset text encoder can refer to a text encoder that has been randomly initialized, so as to... Each occlusion data point is used as input to a pre-defined text encoder for training a single training batch. The corresponding text data can be used as the label data for training. Each text data is input into a preset text encoder, and the output is the reconstructed data of the corresponding text data. That is, the reconstructed data is determined according to the parameters of the current preset text encoder. The reconstruction loss can be used to supervise the similarity between the reconstructed data and the text data, so that the reconstructed data becomes closer and closer to the text data, i.e. the label data, as the training process progresses, thereby achieving the task of reconstructing text data.

[0040] Based on the reconstruction loss, the gradient descent method is used to initially train the preset text encoder until the reconstruction loss converges to meet the preset condition, thus obtaining the initially trained text encoder. In this embodiment, the preset condition is set to the point that the reconstruction loss no longer converges.

[0041] In this embodiment, the text encoder is initially trained by reconstructing occluded data into text data. Only when the pre-trained text encoder learns the scene information to which the text data belongs can the text be accurately reconstructed. This improves the feature extraction capability of the pre-trained text encoder, thereby improving the accuracy of subsequent data sorting.

[0042] The above will Each text data point is occluded separately to obtain... Occlusion data corresponding to text data, based on Text data and The process involves using occlusion data corresponding to text data to train a pre-defined text encoder, resulting in a pre-trained text encoder. This process utilizes occlusion data to guide the text encoder in learning the scene information to which the text data belongs, thereby improving the feature representation ability of the extracted occlusion features and ultimately enhancing the feature representation ability of the text data.

[0043] Step S203, will Each text data point is concatenated with its corresponding first segmentation information, and then input into a pre-trained text encoder for feature extraction. Each text feature vector corresponds to a text data point.

[0044] The first distribution information can refer to the distribution processing information corresponding to the text data. The distribution processing information can include the distribution unit, processing time, etc., and the text feature vector can be used to characterize the feature information of the text data.

[0045] Specifically, the first allocation information can be determined based on the allocation information corresponding to the graphic data, or it can be determined based solely on the allocation information corresponding to the text data. Here, it is taken into consideration that the acquired graphic data may only include text data, so as to ensure that each text data has corresponding allocation information.

[0046] Optionally, to Each text data point is concatenated with its corresponding first segmentation information, and then input into a pre-trained text encoder for feature extraction. Each text feature vector corresponding to the text data includes: For any text data, obtain the first segmentation information of the text data, and concatenate the text data and the first segmentation information to obtain the first concatenation result; The first concatenation result is input into the pre-trained text encoder for feature extraction to obtain the text feature vector of the text data. Traversal From the text data, we get Each text feature vector corresponds to a text data point.

[0047] The first splicing result can refer to the splicing result of text data and the first distribution information, and splicing can refer to the concatenation operation.

[0048] Specifically, each text data has its corresponding first segmentation information, thus each text data corresponds to a first concatenation result. Each first concatenation result undergoes feature extraction through a pre-trained text encoder, which yields... Each text feature vector corresponds to a text data point.

[0049] In this embodiment, text data and distribution information are concatenated, enabling the model to learn the distribution information during text feature extraction. This results in a text feature vector that is more suitable for data distribution when applied to data distribution tasks, thereby improving the accuracy of subsequent data distribution.

[0050] The above will Each text data point is concatenated with its corresponding first distribution information, and then input into a pre-trained text encoder for feature extraction. The step of generating text feature vectors corresponding to text data provides additional information for the data distribution task through distribution information, making the text feature extraction process more inclined to extract text feature vectors that can be used for the data distribution task, thereby improving the adaptability of text feature vectors to the data distribution task and thus improving the accuracy of data distribution.

[0051] Step S204, will Each image data point is concatenated with its corresponding second segmentation information, and then input into a preset image encoder for feature extraction, resulting in... Each image feature vector corresponds to a specific image data point.

[0052] The second sorting information can refer to the sorting processing information corresponding to the image data. The sorting processing information can also include sorting units, processing time, etc. The image feature vector can be used to characterize the feature information of the image data. The preset image encoder can refer to the pre-trained image encoder. The pre-trained image encoder can be an encoder such as the ViT model encoder or the Mask-RCNN model encoder.

[0053] Specifically, the second allocation information can be determined based on the allocation information corresponding to the text and image data, or it can be determined based solely on the allocation information corresponding to the image data. Here, it is taken into consideration that the acquired text and image data may only include image data, so as to ensure that each image data has corresponding allocation information.

[0054] When pre-training the preset image encoder, image data from the task allocation scenario can be used as the pre-training dataset. For example, in the sanitation task allocation scenario, road image data containing cigarette butts and river image data containing plastic bags can be used as pre-training image data. The downstream task during pre-training can be image classification task, semantic segmentation task, etc. In this embodiment, in order to improve the efficiency of pre-training, the preset image encoder is pre-trained using the image classification task. That is, after the preset image encoder is pre-trained, a fully connected layer for image classification is connected. At this time, the pre-training data needs to be labeled. The labels can be road images containing garbage, river images containing garbage, road images without garbage, river images without garbage, etc. The image classification task can be supervised using the cross-entropy loss function.

[0055] Optionally, to Each image data point is concatenated with its corresponding second segmentation information, and then input into a preset image encoder for feature extraction, resulting in... Each corresponding image feature vector includes: For any image data, obtain the second segmentation information of the image data, and stitch the image data and the second segmentation information together to obtain the second stitching result; The second stitching result is input into the image encoder for feature extraction to obtain the image feature vector of the image data. Traversal From image data, we obtain Each image feature vector corresponds to a specific image data point.

[0056] The second stitching result can refer to the stitching result of image data and second-level information, and stitching can also refer to the connection operation.

[0057] Specifically, each image data has its corresponding second segmentation information, thus each image data corresponds to a second stitching result. Each second stitching result undergoes feature extraction through a preset image encoder, which yields... Each image feature vector corresponds to a specific image data point.

[0058] In this embodiment, image data and distribution information are concatenated, enabling the model to learn the distribution information during image feature extraction. This results in a more suitable image feature vector for data distribution when applied to data distribution tasks, thereby improving the accuracy of subsequent data distribution.

[0059] The above will Each image data point is concatenated with its corresponding second segmentation information, and then input into a preset image encoder for feature extraction, resulting in... The step of generating image feature vectors corresponding to image data provides additional information for the image data in the data allocation task through allocation information. This makes the image feature extraction process more inclined to extract image feature vectors that can be used for the data allocation task, thereby improving the adaptability of image feature vectors to the data allocation task and thus improving the accuracy of data allocation.

[0060] Step S205, calculate the sum of each text feature vector and... The inner product between each image feature vector is used to calculate the feature extraction loss based on all the calculated inner products and the preset loss function. The pre-trained text encoder is then retrained based on the feature extraction loss to obtain the final trained text encoder.

[0061] The inner product can be calculated by the dot product between vectors, the preset loss function can be the cross-entropy loss function, and the finally trained text encoder can be a text encoder that can be applied to the actual task allocation scenario.

[0062] Specifically, since a small amount of text and image data is not enough to guide the model to learn the most effective image features, direct training may lead to overfitting. In addition, the image encoder has been pre-trained on a large amount of image data of task allocation scenarios and already has good image encoding capabilities. Moreover, task allocation tasks usually focus more on the representation of text data. In this embodiment, only the text encoder is trained, that is, the parameters of the image encoder are fixed and are not adjusted during the training process.

[0063] Optionally, calculate each text feature vector and... The feature extraction loss is calculated based on the inner product of each image feature vector and a preset loss function, including: For any text feature vector, calculate the sum of the text feature vectors. The inner product between the image feature vectors yields The result of the first inner product calculation; The image feature vector corresponding to the image data that belongs to the same set of image and text data as the text feature vector is determined as the first positive example vector, and the inner product of the text feature vector and the first positive example vector is used as the first positive example calculation result. The image feature vector corresponding to the image data that does not belong to a set of image and text data is determined as the first negative example vector, and the inner product of the text feature vector and each first negative example vector is used as the first negative example calculation result. Based on the calculation results of the first positive example, the calculation results of each first negative example, and the loss function, the first feature extraction sub-loss is calculated, and then iterated through... 1 text feature vector, to obtain The first feature extraction sub-loss is determined. The sum of the losses of each first feature extraction sub-loss is the first feature extraction loss.

[0064] Wherein, the first inner product calculation result can refer to the dot product calculation result between the text feature vector and the image feature vector; the first positive example vector can refer to the image feature vector that has a corresponding relationship with the text feature vector; the corresponding relationship can refer to the text feature vector and the image feature vector corresponding to the same item to be allocated; the first positive example calculation result can refer to the dot product calculation result between the text feature vector and the image feature vector that have a corresponding relationship; the first negative example vector can refer to the image feature vector that does not have a corresponding relationship with the text feature vector; the first negative example calculation result can refer to the dot product calculation result between the text feature vector and the image feature vector that do not have a corresponding relationship; the first feature extraction sub-loss can refer to the feature extraction loss corresponding to the targeted text feature vector; and the first feature extraction loss can refer to the feature extraction loss of all text feature vectors.

[0065] Specifically, for The data consists of a set of text and image data, where each text data point has a corresponding image data point and a non-corresponding image data point. Given a set of image data, let the text feature vector of a text data be sufficiently similar to the image feature vector of its corresponding image data, and not correspond to it. If the image feature vectors of two image data are sufficiently dissimilar, the text feature vectors and image feature vectors corresponding to the same information can be aligned.

[0066] In this embodiment, the feature extraction loss is constructed by calculating the inner product of the image with respect to the text, so that the image and text data corresponding to the same data distribution information have similar feature vector expressions. This allows for fine-tuning of the text encoder to provide text feature vectors that are closer to the final data distribution task, thereby improving the accuracy of data distribution.

[0067] Optionally, calculate each text feature vector and... The feature extraction loss is calculated based on the inner product between each image feature vector and a preset loss function. This also includes: For any image feature vector, calculate the sum of the image feature vector and the sum of ... The inner product of the text feature vectors yields... The result of the second inner product calculation; The text feature vector corresponding to the text data that belongs to the same set of image and text data as the image feature vector is determined as the second positive example vector, and the inner product of the image feature vector and the second positive example vector is used as the calculation result of the second positive example. The text feature vector corresponding to the text data that does not belong to a set of image and text data is determined as the second negative example vector. The inner product of the text feature vector and each second negative example vector is used as the calculation result of the second negative example. Based on the calculation results of the second positive example, the calculation results of each second negative example, and the loss function, the second feature extraction sub-loss is calculated, and then iterated through... From the image feature vectors, we obtain The second feature extraction sub-loss is determined. The sum of the losses of each second feature extraction sub-loss is the second feature extraction loss; The sum of the first feature extraction loss and the second feature extraction loss is used as the feature extraction loss.

[0068] The second inner product calculation result can refer to the dot product calculation result between the image feature vector and the text feature vector. The second positive example vector can refer to the text feature vector that has a corresponding relationship with the image feature vector. The second positive example calculation result can refer to the dot product calculation result between the image feature vector and the text feature vector that have a corresponding relationship. The second negative example vector can refer to the text feature vector that does not have a corresponding relationship with the image feature vector. The second negative example calculation result can refer to the dot product calculation result between the image feature vector and the text feature vector that do not have a corresponding relationship. The second feature extraction sub-loss can refer to the feature extraction loss corresponding to the image feature vector. The second feature extraction loss can refer to the feature extraction loss of all image feature vectors. The feature extraction loss can refer to the final loss used for model training.

[0069] Specifically, for The dataset consists of a set of image and text data, where each image has a corresponding text data item and a non-corresponding text data item. Given a set of text data, let the image feature vector of an image data be sufficiently similar to the text feature vector of its corresponding text data, and let them not correspond to each other. If the text feature vectors of the text data are sufficiently dissimilar, the image feature vectors and text feature vectors corresponding to the same information can be aligned.

[0070] Since the position of vectors affects the result of the dot product operation, bidirectional calculation is required, that is, calculating both the inner product of text feature vectors and image feature vectors, and the inner product of image feature vectors and text feature vectors.

[0071] In this embodiment, the feature extraction loss is constructed by calculating the inner product of text with respect to the image, so that the text and image data corresponding to the same data distribution information have similar feature vector expressions. This allows for fine-tuning of the text encoder to provide text feature vectors that are closer to the final data distribution task, thereby improving the accuracy of data distribution.

[0072] The above calculations for each text feature vector are respectively... The inner product between image feature vectors is used to calculate the feature extraction loss based on all the calculated inner products and the preset loss function. The pre-trained text encoder is then retrained based on the feature extraction loss to obtain the final trained text encoder. The text feature vectors and image feature vectors with corresponding relationships are supervised to ensure that the text and image data corresponding to the same allocation information have similar feature vector expressions, thereby improving the accuracy of subsequent data allocation.

[0073] Step S206: Determine the final trained text encoder and image encoder to form a trained data distribution model.

[0074] The trained data distribution model can be used for data distribution tasks in corresponding task distribution scenarios.

[0075] Specifically, the trained data distribution model is connected to a pre-trained downstream task classifier. That is, the outputs of the finally trained text encoder and image encoder are concatenated and then input into the pre-trained downstream task classifier. The pre-trained downstream task classifier can be a classifier used for task distribution. The output of the classifier can be the unit category to which the task is distributed. For example, in the scenario of government affairs task distribution, the unit category can be urban management unit, finance unit, etc.

[0076] The steps described above, which determine the final trained text encoder and image encoder to form a trained data distribution model, result in a data distribution model that is more adapted to the task distribution scenario, thereby improving the accuracy of data distribution.

[0077] In this embodiment, the text encoder is initially trained using text data that matches the scene, enabling the pre-trained text encoder to learn scene information. The text encoder is then retrained by combining the sorting information and text data, and image feature vectors are introduced as a reference. This allows the text encoder to learn to align text feature vectors and image feature vectors belonging to the same sorting information while learning the sorting information, thereby improving the expressive power of the features extracted by the data sorting model and thus improving the accuracy of data sorting.

[0078] See Figure 3 This is a flowchart illustrating a data distribution method provided in Embodiment 2 of the present invention. The data distribution method includes the following steps: Step S301: Obtain the data to be distributed. When the data to be distributed belongs to image and text data, input the data to be distributed into the trained data distribution model for feature extraction to obtain image and text feature vectors. Step S302: Search the image and text feature vectors in the preset database to obtain at least one image and text storage vector. Step S303: The image and text feature vectors are concatenated with each image and text storage vector, and the concatenation results are input into the preset fully connected layer for similarity calculation to obtain the similarity score of the corresponding image and text storage vectors. Step S304: Determine the image and text storage vector corresponding to the maximum value among all similarity scores as the reference vector, and use the distribution information corresponding to the reference vector as the distribution information of the data to be distributed.

[0079] The trained data allocation model is obtained based on the data allocation model training method described in Example 1. The data to be allocated refers to the data related to the matters requiring data allocation. The image / text feature vector represents the feature information of the data to be allocated. The pre-set database may include multiple stored image / text vectors corresponding to different image / text information. The similarity score can be used to represent the degree of similarity between the image / text feature vector and the stored image / text vector. The pre-set fully connected layer can be used to predict the similarity score between input vectors. The reference vector refers to the stored image / text vector used to provide allocation information for the data to be allocated.

[0080] Specifically, when the data to be distributed is image data, feature extraction can be performed using the image encoder in the trained data distribution model to obtain the target image feature vector. Then, the scorescript search method in the Elasticsearch architecture is used to search for similar historical events in the target image feature vector. The distribution information of the image data is determined based on the distribution information corresponding to the most similar historical event in the search results.

[0081] When the data to be distributed is text data, the text data can be input into the final trained text encoder in the trained data distribution model for feature extraction to obtain the target text feature vector. The target text feature vector is then searched for similar historical events in the preset database. The distribution information of the text data is determined based on the distribution information corresponding to the most similar historical event in the search results.

[0082] In one implementation, when the data to be distributed is text data, Elasticsearch's text matching algorithm can be used to recall historical events similar to the text data, and the distribution information of the text data can be determined based on the distribution information corresponding to the most similar historical event in the recall results.

[0083] In this embodiment, different distribution processing methods are determined for multimodal data to realize the search and retrieval of historical events of multimodal data, thereby improving the generalization ability of data distribution tasks. The adoption of corresponding distribution processing methods also improves the accuracy of data distribution.

[0084] The model training method corresponding to the data allocation in Example 1 above, Figure 4 A structural block diagram of the data distribution model training device provided in Embodiment 3 of the present invention is shown. This data distribution model training device is applied to a client. The computer device corresponding to the client connects to a corresponding server to obtain text and image data to be processed from the server. The data to be processed may refer to data that needs to be distributed. The computer device corresponding to the client is equipped with a preset text encoder and a preset image encoder. The preset text encoder may be a randomly initialized text encoder, and the preset image encoder may be a pre-trained image encoder. For ease of explanation, only the parts relevant to the embodiments of the present invention are shown.

[0085] See Figure 4 The model training device for this data distribution includes: Data acquisition module 41 is used to acquire The dataset consists of groups of image and text data to be processed. Each group includes one image and one text data. Integers greater than zero; Text occlusion module 42, used to... Each text data point is occluded separately to obtain... Occlusion data corresponding to text data, based on Text data and Using occlusion data corresponding to text data, a pre-set text encoder is trained to obtain a pre-trained text encoder. The first feature extraction module 43 is used to extract features from the first feature extraction module 43. Each text data point is concatenated with its corresponding first distribution information, and then input into a pre-trained text encoder for feature extraction. Each corresponding text feature vector; The second feature extraction module 44 is used to extract the features. Each image data point is concatenated with its corresponding second segmentation information, and then input into a preset image encoder for feature extraction, resulting in... Image feature vectors corresponding to each image data; Model training module 45 is used to calculate the sum of each text feature vector and... The inner product between each image feature vector is used to calculate the feature extraction loss based on all the calculated inner products and the preset loss function. The pre-trained text encoder is then retrained based on the feature extraction loss to obtain the final trained text encoder. The model determination module 46 is used to determine the final trained text encoder and image encoder to form a trained data distribution model.

[0086] Optionally, the text occlusion module 42 mentioned above includes: The occlusion feature extraction unit is used to extract occlusion features from occlusion data of any corresponding text data by inputting the occlusion data into a preset text encoder. The feature reconstruction unit is used to input the occluded features into a preset text encoder for feature reconstruction to obtain reconstructed data; The reconstruction sub-loss unit is used to calculate the reconstruction sub-loss based on the text data corresponding to the reconstructed data and occlusion data and the preset reconstruction loss function; The initial training unit is used to traverse the occlusion data of each corresponding text data to obtain the reconstruction sub-loss of the corresponding text data. The reconstruction sub-losses of all corresponding text data are added together to obtain the reconstruction loss. Based on the reconstruction loss, the preset text encoder is initially trained to obtain the pre-trained text encoder.

[0087] Optionally, the first feature extraction module 43 mentioned above includes: The first splicing unit is used to obtain the first distribution information of any text data, splice the text data and the first distribution information to obtain the first splicing result. The text feature extraction unit is used to input the first concatenation result into the pre-trained text encoder for feature extraction, and obtain the text feature vector of the text data. Text data traversal unit, used for traversal From the text data, we get Each text feature vector corresponds to a text data point.

[0088] Optionally, the second feature extraction module 44 mentioned above includes: The second stitching unit is used to obtain the second segmentation information of any image data, and stitch the image data and the second segmentation information together to obtain the second stitching result. The image feature extraction unit is used to input the second stitching result into the image encoder for feature extraction, and obtain the image feature vector of the image data. Image data traversal unit, used for traversal From image data, we obtain Each image feature vector corresponds to a specific image data point.

[0089] Optionally, the above model training module 45 includes: The first inner product calculation unit is used to calculate the sum of the text feature vector and the inner product for any given text feature vector. The inner product between the image feature vectors yields The result of the first inner product calculation; The first positive example determination unit is used to determine the image feature vector corresponding to the image data that belongs to the same set of image and text data as the text data corresponding to the text feature vector as the first positive example vector, and to use the inner product of the text feature vector and the first positive example vector as the first positive example calculation result. The first negative example determination unit is used to determine the image feature vector corresponding to the image data that does not belong to a set of image and text data as the first negative example vector, and to use the inner product calculation result of the text feature vector and each first negative example vector as the first negative example calculation result. The first loss calculation unit is used to calculate the first feature extraction sub-loss based on the calculation results of the first positive example, the calculation results of each first negative example, and the loss function, and iterates through... 1 text feature vector, to obtain The first feature extraction sub-loss is determined. The sum of the losses of each first feature extraction sub-loss is the first feature extraction loss.

[0090] Optionally, the model training module 45 mentioned above also includes: The second inner product calculation unit is used to calculate the sum of the image feature vector and the inner product for any given image feature vector. The inner product of the text feature vectors yields... The result of the second inner product calculation; The second positive example determination unit is used to determine the text feature vector corresponding to the text data that belongs to the same set of image and text data as the image feature vector as the second positive example vector, and to use the inner product of the image feature vector and the second positive example vector as the second positive example calculation result. The second negative example determination unit is used to determine the text feature vector corresponding to the text data that does not belong to a set of image and text data as the second negative example vector, and to use the inner product calculation result of the text feature vector and each second negative example vector as the second negative example calculation result. The second loss calculation unit is used to calculate the second feature extraction sub-loss based on the calculation results of the second positive examples, the calculation results of each second negative example, and the loss function, and iterates through... From the image feature vectors, we obtain The second feature extraction sub-loss is determined. The sum of the losses of each second feature extraction sub-loss is the second feature extraction loss; The loss summation unit is used to use the sum of the first feature extraction loss and the second feature extraction loss as the feature extraction loss.

[0091] It should be noted that the information interaction and execution process between the above modules and units are based on the same concept as the method embodiments of the present invention. For details on their specific functions and technical effects, please refer to the method embodiments section, which will not be repeated here.

[0092] Figure 5 This is a schematic diagram of the structure of a computer device provided in Embodiment 4 of the present invention. Figure 5 As shown, the computer device of this embodiment includes: at least one processor (in...) Figure 5 Only one is shown in the diagram), a memory, and a computer program stored in the memory that can run on at least one processor, wherein the processor can execute the computer program to implement the steps in the model training method embodiments of any of the above data distributions.

[0093] This computer device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that... Figure 5 The examples of computer devices are merely examples and do not constitute a limitation on computer devices. Computer devices may include more or fewer components than shown in the illustration, or combinations of certain components, or different components, such as network interfaces, displays, and input devices.

[0094] The processor referred to can be a CPU, but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any other conventional processor.

[0095] The memory can include readable storage media, internal memory, etc., wherein the internal memory can be the RAM of a computer device, providing an environment for the operation of the operating system and computer-readable instructions stored in the readable storage media. The readable storage media can be the hard drive of the computer device, or in other embodiments, it can be an external storage device of the computer device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory can include both internal storage units and external storage devices of the computer device. The memory can be used to store the operating system, applications, bootloader, data, and other programs, such as program code for computer programs. The memory can also be used to temporarily store data that has been output or will be output.

[0096] Those skilled in the art will understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the functions described above can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this invention. The specific working process of the units and modules in the above device can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention can implement all or part of the processes in the methods of the above embodiments by instructing related hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the above method embodiments. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. A computer-readable medium can include at least: any entity or device capable of carrying computer program code, a recording medium, a computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0097] The present invention can implement all or part of the processes in the methods of the above embodiments, or it can be accomplished by a computer program product. When the computer program product is run on a computer device, the computer device executes the steps in the above method embodiments.

[0098] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0099] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0100] In the embodiments provided by this invention, it should be understood that the disclosed apparatus / computer devices and methods can be implemented in other ways. For example, the apparatus / computer device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0101] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0102] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A data distribution model training method, characterized in that, The model training method includes: Get The dataset consists of groups of image and text data to be processed. Each group includes one image and one text data. Integers greater than zero; Will Each text data point is occluded separately to obtain... Occlusion data corresponding to text data, according to the The text data and the above Using occlusion data corresponding to text data, a pre-set text encoder is trained to obtain a pre-trained text encoder. The Each text data point is concatenated with its corresponding first distribution information, and then input into the pre-trained text encoder for feature extraction. Each corresponding text feature vector; Will Each image data point is concatenated with its corresponding second segmentation information, and then input into a preset image encoder for feature extraction, resulting in... Image feature vectors corresponding to each image data; Calculate the sum of each text feature vector. The inner product between each image feature vector is used to calculate the feature extraction loss based on all the calculated inner products and the preset loss function. The pre-trained text encoder is then retrained based on the feature extraction loss to obtain the final trained text encoder. The final trained text encoder and the image encoder are used to form a trained data distribution model.

2. The model training method according to claim 1, characterized in that, According to the The text data and the Using occlusion data corresponding to the text data, a pre-defined text encoder is trained to obtain a pre-trained text encoder, including: For any occlusion data corresponding to text data, the occlusion data is input into the preset text encoder for feature extraction to obtain occlusion features; The occlusion features are input into a preset text encoder for feature reconstruction to obtain reconstructed data; Based on the reconstructed data, the text data corresponding to the occlusion data, and the preset reconstruction loss function, calculate the reconstruction sub-loss; By traversing the occlusion data of each corresponding text data, the reconstruction sub-loss of the corresponding text data is obtained. The reconstruction sub-losses of all corresponding text data are added together to obtain the reconstruction loss. Based on the reconstruction loss, the preset text encoder is initially trained to obtain the initially trained text encoder.

3. The model training method according to claim 1, characterized in that, The Each text data point is concatenated with its corresponding first distribution information, and then input into the pre-trained text encoder for feature extraction. Each text feature vector corresponding to the text data includes: For any text data, obtain the first distribution information of the text data, and concatenate the text data and the first distribution information to obtain a first concatenation result; The first concatenation result is input into the pre-trained text encoder for feature extraction to obtain the text feature vector of the text data. Traversing the From the text data, the following is obtained: Each text feature vector corresponds to a text data point.

4. The model training method according to claim 1, characterized in that, The Each image data point is concatenated with its corresponding second segmentation information, and then input into a preset image encoder for feature extraction, resulting in... Each corresponding image feature vector includes: For any image data, obtain the second segmentation information of the image data, and stitch the image data and the second segmentation information together to obtain a second stitching result; The second stitching result is input into the image encoder for feature extraction to obtain the image feature vector of the image data; Traversing the From the image data, the following is obtained: Each image feature vector corresponds to a specific image data point.

5. The model training method according to claim 1, characterized in that, The calculation of each text feature vector and The feature extraction loss is calculated based on the inner product of each image feature vector and a preset loss function, including: For any text feature vector, calculate the inner product between the text feature vector and each of the N image feature vectors to obtain... The result of the first inner product calculation; The image feature vector corresponding to the image data that belongs to the same set of image and text data as the text feature vector is determined as the first positive example vector, and the inner product of the text feature vector and the first positive example vector is used as the first positive example calculation result. The image feature vector corresponding to the image data that does not belong to a set of image and text data is determined as the first negative example vector, and the inner product of the text feature vector and each first negative example vector is used as the first negative example calculation result. Based on the calculation results of the first positive example, the calculation results of each first negative example, and the loss function, the first feature extraction sub-loss is calculated, and then iterated through... 1 text feature vector, obtain The first feature extraction sub-loss is used to determine the... The sum of the losses of each first feature extraction sub-loss is the first feature extraction loss.

6. The model training method according to claim 5, characterized in that, The calculation of each text feature vector and The inner product between image feature vectors is used to calculate the feature extraction loss based on all the calculated inner products and a preset loss function. This also includes: For any image feature vector, calculate the image feature vector and the... The inner product of the text feature vectors yields... The result of the second inner product calculation; The text feature vector corresponding to the text data that belongs to the same set of image and text data as the image feature vector is determined as the second positive example vector, and the inner product of the image feature vector and the second positive example vector is used as the second positive example calculation result; The text feature vector corresponding to the text data that does not belong to a set of image and text data is determined as the second negative example vector, and the inner product of the text feature vector and each second negative example vector is used as the second negative example calculation result. Based on the calculation results of the second positive example, the calculation results of each second negative example, and the loss function, the second feature extraction sub-loss is calculated, and then iterated through... From the image feature vectors, we obtain The second feature extraction sub-loss is used to determine the... The sum of the losses of each second feature extraction sub-loss is the second feature extraction loss; The sum of the first feature extraction loss and the second feature extraction loss is used as the feature extraction loss.

7. A data distribution method, the data distribution method comprising: Acquire the data to be distributed. When the data to be distributed belongs to image and text data, input the data to be distributed into the trained data distribution model for feature extraction to obtain image and text feature vectors. The image and text feature vectors are retrieved from a preset database to obtain at least one image and text storage vector. The image and text feature vectors are concatenated with each image and text storage vector, and the concatenation results are input into a preset fully connected layer for similarity calculation to obtain the similarity score of the corresponding image and text storage vector. The image and text storage vector corresponding to the maximum value among all similarity scores is determined as the reference vector, and the distribution information corresponding to the reference vector is used as the distribution information of the data to be distributed. The trained data distribution model is obtained based on the data distribution model training method according to any one of claims 1-6.

8. A data distribution model training device, characterized in that, The model training device includes: The data acquisition module is used to acquire... The dataset consists of groups of image and text data to be processed. Each group includes one image and one text data. Integers greater than zero; Text masking module, used to... Each text data point is occluded separately to obtain... Occlusion data corresponding to text data, according to the The text data and the Using occlusion data corresponding to text data, a pre-set text encoder is trained to obtain a pre-trained text encoder. The first feature extraction module is used to extract the features of the first feature extraction module. Each text data point is concatenated with its corresponding first distribution information, and then input into the pre-trained text encoder for feature extraction. Each corresponding text feature vector; The second feature extraction module is used to extract features from the second feature extraction module. Each image data point is concatenated with its corresponding second segmentation information, and then input into a preset image encoder for feature extraction, resulting in... Image feature vectors corresponding to each image data; The model training module is used to calculate the sum of each text feature vector and... The inner product between each image feature vector is used to calculate the feature extraction loss based on all the calculated inner products and the preset loss function. The pre-trained text encoder is then retrained based on the feature extraction loss to obtain the final trained text encoder. The model determination module is used to determine the final trained text encoder and the image encoder to form a trained data distribution model.

9. A computer device, characterized in that, The computer device includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the model training method as described in any one of claims 1 to 6.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the model training method as described in any one of claims 1 to 6.

Images