Corpus processing method, device, electronic device, storage medium and product

Abstract

The present application discloses a corpus processing method, device, electronic device, storage medium and product. The method comprises: in response to a corpus processing instruction, determining the corresponding corpus to be processed; wherein the corpus to be processed comprises at least one of the following: graphics to be processed and images to be processed; taking the corpus to be processed as input, obtaining corresponding fusion features using a target modality fusion model, and performing business processing based on the fusion features. The embodiment of the present application utilizes a target modality fusion model to process corpus, and the fusion features outputted by the model are content features that can reflect the feature relationship between modalities, thereby improving the reliability and effectiveness of content feature extraction, and thereby improving the accuracy of content processing scenarios.

Description

Technical Field

[0001] The present application relates to the field of Internet communication technology, and in particular to a corpus processing method, device, electronic device, storage medium and product. Background Art

[0002] With the rapid development of Internet communication technology, various Internet products emerge in an endless stream. Taking Internet products that provide content (such as pictures and text) experience as an example, the content often has more than one modality. In related technologies, when using related models to extract content features, the obtained content features are often a simple splicing of common features of various modalities, and cannot provide content features that can reflect the feature relationship between modalities. Therefore, it is necessary to provide a content processing solution that can output such content features (content features that reflect the feature relationship between modalities). Summary of the invention

[0003] In order to solve the problem that the existing technology cannot obtain content features that can reflect the feature relationship between modalities when extracting content features, the present application provides a corpus processing method, device, electronic device, storage medium and product:

[0004] According to a first aspect of the present application, a corpus processing method is provided, the method comprising:

[0005] In response to the corpus processing instruction, the corresponding corpus to be processed is determined; wherein the corpus to be processed includes at least one of the following: graphics to be processed and images to be processed;

[0006] The corpus to be processed is used as input, and a target modal fusion model is used to obtain corresponding fusion features, so as to perform business processing based on the fusion features; wherein the target modal fusion model is obtained by performing machine learning training on multiple image-text pair samples and adjusting the parameters of a preset model during the training process, each of the image-text pair samples indicates a paired image sample and a text sample, and the training process includes processing an image-text feature set based on preset task information, the image-text feature set includes multiple image-text features, and the multiple image-text features correspond one-to-one to the multiple image-text pair samples, and the preset task information indicates a task of learning two types of correlations in the image-text feature set, and the two types of correlations are respectively a first correlation between the image feature part and the text feature part in the same image-text feature in the image-text feature set, and a second correlation between the image feature part and the text feature part between different image-text features.

[0007] According to a second aspect of the present application, a corpus processing device is provided, the device comprising:

[0008] A response module: used to respond to the corpus processing instruction and determine the corresponding corpus to be processed; wherein the corpus to be processed includes at least one of the following: graphics and text to be processed and images to be processed;

[0009] Output module: used for taking the corpus to be processed as input, obtaining corresponding fusion features by using the target modality fusion model, so as to perform business processing based on the fusion features; wherein the target modality fusion model is obtained by performing machine learning training on multiple image-text pair samples and adjusting the parameters of the preset model during the training process, each of the image-text pair samples indicates a paired image sample and a text sample, the training process includes processing an image-text feature set based on preset task information, the image-text feature set includes multiple image-text features, the multiple image-text features correspond one-to-one to the multiple image-text pair samples, the preset task information indicates a task of learning two types of correlations in the image-text feature set, the two types of correlations are respectively a first correlation between the image feature part and the text feature part in the same image-text feature in the image-text feature set, and a second correlation between the image feature part and the text feature part between different image-text features.

[0010] According to a third aspect of the present application, an electronic device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the corpus processing method as described in the first aspect.

[0011] According to a fourth aspect of the present application, a computer-readable storage medium is provided, in which at least one instruction or at least one program is stored, and the at least one instruction or the at least one program is loaded and executed by a processor to implement the corpus processing method as described in the first aspect.

[0012] According to a fifth aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the corpus processing method as described in the first aspect.

[0013] The present application provides a corpus processing method, device, electronic device, storage medium and product, which have the following technical effects:

[0014] The present application responds to the corpus processing instruction and determines the corresponding corpus to be processed; then, the corpus to be processed is used as input, and the corresponding fusion features are obtained by using the target modal fusion model to perform business processing based on the fusion features. The present application uses the target modal fusion model to process the corpus, and the fusion features outputted by the target modal fusion model are content features that can reflect the feature relationship between modalities, thereby improving the reliability and effectiveness of content feature extraction, and thus improving the accuracy of content processing scenarios. The present application provides a model architecture that first extracts image / text features and then performs modal fusion. Each of the multiple image-text pair samples as the current batch sample set includes an image sample and a text sample; at the same time, the multiple image-text features constituting the image-text feature set correspond one-to-one to the above-mentioned multiple image-text pair samples, and each image-text feature is composed of an image feature corresponding to an image sample indicating the same image-text pair sample and a text feature corresponding to a text sample. The correlation between the image feature part and the text feature part in the same graphic feature can be used as the "positive sample" to be learned, and the correlation between the image feature part and the text feature part between different graphic features can be used as the "negative sample" to be learned. The present application provides positive and negative samples for learning for the training of the target modality fusion model, ensuring the training effect. At the same time, the construction of "negative samples" no longer relies on manual work and uses a batch of sample sets as input data, which improves the training efficiency and ensures that the training obtains a target modality fusion model with high generalization ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0016] Figure 1 is a schematic diagram of an application environment provided by an embodiment of the present application;

[0017] Figure 2 It is a flowchart of a corpus processing method provided in an embodiment of the present application;

[0018] Figure 3 It is a flow chart of training to obtain a target modality fusion model provided in an embodiment of the present application;

[0019] Figure 4 It is a flow chart of using a third preset model to process a graphic feature set based on preset task information to obtain an actual processing result provided by an embodiment of the present application;

[0020] Figure 5It is also a flow chart of training to obtain a target modality fusion model provided in an embodiment of the present application;

[0021] Figure 6 is an application schematic diagram of the target modality fusion model provided in an embodiment of the present application;

[0022] Figure 7 It is also a schematic diagram of the application of the target modality fusion model provided in the embodiment of the present application;

[0023] Figure 8 is a schematic diagram of a content processing system provided by an embodiment of the present application;

[0024] Fig. 9 It is a block diagram of a corpus processing device provided in an embodiment of the present application;

[0025] Fig.10 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0027] It should be noted that the terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0028] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0029] PGC (Professional Generated Content): refers to content produced by professional users and experts. It is also called PPC (Professionally-produced Content).

[0030] UGC (User Generated Content): refers to user-generated content.

[0031] PUGC (Professional User Generated Content): is professional content produced in the form of UGC that is relatively close to PGC.

[0032] MCN (Multi-Channel Network) is a product form of a multi-channel network that unites PGC content and, with the strong support of capital, ensures the continuous output of content, thereby ultimately achieving stable commercial realization.

[0033] Feeds (web feed, news feed, syndicated feed): a source of information, also translated as source material, feed, information provision, feed, summary, source, news subscription, network source is a data format. Websites use it to spread the latest information to users, usually arranged in a timeline. Timeline is the most primitive, direct and basic display form of Feed. The prerequisite for users to subscribe to a website is that the website provides a source of information. The convergence of feeds in one place is called aggregation, and the software used for aggregation is called an aggregator. For end users, an aggregator is software specifically used to subscribe to websites, generally also called RSS (Really Simple Syndication) readers, feed readers, news readers, etc.

[0034] Transformer: It is a model based on the encoder-decoder structure, which mainly consists of two parts: encoder and decoder. It uses the self-attention mechanism for machine translation, has achieved good results, and has a fast training speed.

[0035] See also Figure 1 , Figure 1It is a schematic diagram of an application environment provided by an embodiment of the present application, and the application environment may include a client 10 and a server 20. The client 10 and the server 20 may be directly or indirectly connected via wired or wireless communication. Relevant objects (such as users, simulators) may send corpus processing instructions to the server 20 via the client 10. The server 20 determines the corresponding corpus to be processed according to the received corpus processing instructions, and then obtains the fusion features corresponding to the corpus to be processed using the target modal fusion model. The target modal fusion model may be obtained by training the server 20 through the following steps: first, the image features corresponding to each image sample are obtained using the first preset model, and the text features corresponding to each text sample are obtained using the second preset model; then, with the image-text feature set as input, the image-text feature set is processed based on the preset task information using the third preset model to obtain the actual processing result; further, the parameters of the third preset model are adjusted according to the actual processing result and the target processing result to meet the model convergence condition; finally, the target modal fusion model is obtained based on the first preset model, the second preset model and the third preset model that meets the model convergence condition. It should be noted that Figure 1 Just an example.

[0036] The client 10 may be a physical device such as a smart phone, a computer (such as a desktop computer, a tablet computer, a laptop computer), an augmented reality (AR) / virtual reality (VR) device, a digital assistant, an intelligent voice interaction device (such as an intelligent speaker), an intelligent wearable device, an intelligent home appliance, a vehicle-mounted terminal, etc., or may be software running in a physical device, such as a computer program. The operating system corresponding to the client may be an Android system (Android system), an iOS system (a mobile operating system developed by Apple), a Linux system (an operating system), a Microsoft Windows system (Microsoft Windows operating system), etc.

[0037] The server end 20 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The server may include a network communication unit, a processor, a memory, etc. The server end may provide background services for the corresponding client.

[0038] In practical applications, the target modality fusion model may not be obtained through training on the server side 20.

[0039] The training scheme or corpus processing scheme of the modal fusion model provided in the embodiments of the present application can use technologies related to cloud computing. Cloud computing is a computing model that distributes computing tasks on a resource pool composed of a large number of computers, enabling various application systems to obtain computing power, storage space and information services as needed. The network that provides resources is called a "cloud". The resources in the "cloud" are infinitely expandable in the eyes of users, and can be obtained at any time, used on demand, expanded at any time, and paid for by use. As a provider of basic cloud computing capabilities, a cloud computing resource pool (referred to as a cloud platform, generally referred to as an IaaS (Infrastructure as a Service) platform) will be established, and various types of virtual resources will be deployed in the resource pool for external customers to choose to use. The cloud computing resource pool mainly includes: computing devices (virtualized machines, including operating systems), storage devices, and network devices. According to the logical function division, the PaaS (Platform as a Service) layer can be deployed on the IaaS (Infrastructure as a Service) layer, and the SaaS (Software as a Service) layer can be deployed on the PaaS layer. SaaS can also be deployed directly on IaaS. PaaS is a platform for software operation, such as databases, web containers, etc. SaaS is a variety of business software, such as web portals, SMS mass senders, etc. Generally speaking, SaaS and PaaS are upper layers relative to IaaS.

[0040] It should be noted that for image samples, text samples, image-text pair samples and corpus to be processed that are associated with user information, when the embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0041] The following describes a specific embodiment of a corpus processing method of the present application. Figure 2 It is a flowchart of a corpus processing method provided in an embodiment of the present application. The present application provides method operation steps as described in the embodiment or flowchart, but may include more or fewer operation steps based on conventional or non-creative labor. The order of steps listed in the embodiment is only one way of executing the steps among many orders, and does not represent the only order of execution. When the actual system or product is executed, it can be executed in sequence or in parallel (for example, in a parallel processor or multi-threaded processing environment) according to the method shown in the embodiment or the accompanying drawings. Specifically, Figure 2As shown, the method may include:

[0042] S201: In response to a corpus processing instruction, determining corresponding corpus to be processed; wherein the corpus to be processed includes at least one of the following: text to be processed and images to be processed;

[0043] In the embodiment of the present application, in response to the corpus processing instruction, the server determines the corresponding corpus to be processed. The corpus processing instruction can be triggered and sent by the client, or it can be triggered and generated by the server. The corpus to be processed is the candidate content to be recommended. After the subsequent steps, the corresponding fusion features are obtained, and the fusion features can be matched with the feature information of the relevant user to determine whether to recommend the content.

[0044] S202: Taking the corpus to be processed as input, obtaining corresponding fusion features using a target modality fusion model, and performing business processing based on the fusion features; wherein the target modality fusion model is obtained by performing machine learning training on multiple image-text pair samples and adjusting the parameters of a preset model during the training process, each of the image-text pair samples indicates a paired image sample and a text sample, the training process includes processing an image-text feature set based on preset task information, the image-text feature set includes multiple image-text features, the multiple image-text features correspond one-to-one to the multiple image-text pair samples, the preset task information indicates a task of learning two types of correlations in the image-text feature set, the two types of correlations are respectively a first correlation between an image feature part and a text feature part in the same image-text feature in the image-text feature set, and a second correlation between an image feature part and a text feature part between different image-text features.

[0045] In an embodiment of the present application, the server side uses the corpus to be processed as input, and uses a target modality fusion model to obtain corresponding fusion features, so as to perform business processing based on the fusion features. In combination with the "corpus to be processed is a candidate content to be recommended" in step S201, the corresponding fusion features are determined, and before executing the matching step, it can also be determined whether it is abnormal content that is not suitable for recommendation based on the fusion features. Content often has more than one modality, and the fusion features extracted by the target modality fusion model are not just a simple perception of multiple modalities, but also establish cross-modal connections. Compared with the simple splicing of common features of various modalities in related technologies, the fusion features extracted in the corpus processing of the embodiment of the present application improve the information granularity of the content features and improve the effect of the business related to content processing. The business related to content processing here can use the content processing system 80 described later.

[0046] For the business processing based on the fusion feature, it can be understood that the fusion feature is used in the relevant business to obtain the business result, and the fusion feature is a contributing feature to obtain the business result of the relevant business. If at least two steps are required to obtain the business result of the relevant business, the fusion feature can participate in at least one of the at least two steps. There can also be other participating features in the step in which the fusion feature participates.

[0047] The related business may be a business related to content processing. Taking the content recommendation business as an example, it is possible to determine whether the corpus to be processed can be used as content to be recommended based on the fusion features corresponding to the corpus to be processed. Taking the content quality detection business as an example, it is possible to determine whether the corpus to be processed can be used as high-quality content based on the fusion features corresponding to the corpus to be processed. Taking the content classification business as an example, it is possible to determine the content classification to which the corpus to be processed belongs based on the fusion features corresponding to the corpus to be processed.

[0048] The business related to content processing can also be subdivided based on the content source. For example, if live video is used as the content source of the corpus to be processed, then the related business may be the business related to live content processing. If the corpus to be processed indicates a product image and text in a live video frame, then it can be determined whether the product corresponding to the product image and text is an abnormal product based on the fusion features corresponding to the corpus to be processed. If online conference video is used as the content source of the corpus to be processed, then the related business may be the business related to online conference content processing. If the corpus to be processed indicates a screenshot, then it can be determined whether the audio and video of the current meeting are synchronized based on the fusion features and audio features corresponding to the corpus to be processed (corresponding to the time segment of the screenshot).

[0049] Content processing services can also be subdivided based on the role of fusion features. For example, in the content model training service, fusion features are used as samples for training the target model, which can be Figure 6 The picture and text content review model and the pure image content review model shown in the figure. In the content collection mining business, the fusion feature is used as the benchmark feature to recall the content corresponding to the feature matching the benchmark feature, and then the content collection is constructed based on the corpus to be processed and the recalled content.

[0050] For the image-text pair sample, it is composed of a paired image sample and a text sample. An image-text pair sample is composed of an image sample and a text sample. An image sample may include at least one image, such as a collection of video frame images. The image indicated by the image sample may be a static image or a dynamic image. The constituent object of a text sample may be a sentence, a paragraph or a discourse, and the constituent object may involve at least one language. Whether an image sample and a text sample are paired can be determined based on whether they have an association relationship. The image sample and the text sample in the image-text pair sample have an association relationship. To a certain extent, the image sample can be used as an image representation of the related text sample, and correspondingly, the text sample can be used as a text representation of the related image sample.

[0051] The following will introduce the process of training to obtain the target modality fusion model, such as Figure 3 As shown, the process includes:

[0052] S301: Taking the plurality of image-text pair samples as input, using a first preset model to obtain image features corresponding to each of the plurality of image samples corresponding to the plurality of image-text pair samples;

[0053] The first preset model is a machine learning model, which can be a machine learning model in training or a machine learning model that has completed training. The first preset model realizes the extraction of image features. The input to the first preset model is a picture-text pair sample. The object of image feature extraction by the first preset model is the image sample in the picture-text pair sample. The image features may include at least one of the following: color features, texture features, shape features, and spatial relationship features of the image samples. It should be noted that the server end executing steps S301-S305 here and the server end executing steps S201-S202 above can be the same server end or different server ends.

[0054] In an exemplary embodiment, taking the multiple image-text pair samples as input and using a first preset model to obtain image features corresponding to each of the multiple image samples corresponding to the multiple image-text pair samples may include the following steps: first, inputting the multiple image-text pair samples into the first preset model; then, using at least two sub-models in the first preset model to sequentially process each of the multiple image samples corresponding to the multiple image-text pair samples to obtain image features corresponding to each of the image samples.

[0055] The first preset model may include at least two sub-models, and the at least two sub-models may include a preset convolutional neural network model (such as a CNN model) set at the input end and a preset machine learning model with a transformation structure (such as a Transformer structure) set at the output end. Multiple image-text pair samples can be input into the first preset model in sequence, and the first preset model processes each image-text pair sample in sequence, and outputs the image features corresponding to the image sample in each image-text pair sample. In the process of the first preset model processing the image-text pair samples, the CNN model close to the input end first extracts the image features of the image-text pair samples to obtain the first feature; then the preset machine learning model with the Transformer structure processes the first feature to obtain the second feature, and the second feature is used as the image feature output by the first preset model. Using the CNN model to extract the first feature can effectively focus on the relatively basic and underlying visual elements in the image sample, and then using the preset machine learning model with the Transformer structure to extract the second feature can integrate these visual elements to capture higher-level visual semantic information, thereby ensuring that the image features output by the first preset model can take into account both visual elements and visual semantic information to achieve more comprehensive feature expression. See Figure 5 , the first feature can be obtained by randomly sampling the feature map.

[0056] Taking into account that the image indicated by the image sample may include text content (such as text watermarks, text on emoticon images), when using the first preset model to extract image features, the text area in the image can be cut out first and then the relevant image feature extraction steps can be performed; or the text content can still be used as the data source of the image features, and the relevant image feature extraction steps can be directly performed without performing the above-mentioned cutting steps.

[0057] The following is an introduction to the acquisition of image and text pair samples:

[0058] The process of obtaining an image-text pair sample may include the following steps: 1) obtaining the image sample in response to an image-text pair sample acquisition instruction; 2) obtaining text associated with the image sample; wherein the data source of the associated text is the description information corresponding to the image sample, the retrieval information corresponding to the image sample, or the text recognition result of the image sample; 3) constructing the image-text pair sample based on the image sample and the associated text; wherein the associated text is a text sample in the image-text pair sample.

[0059] The instruction to obtain the sample of the image-text pair can be generated by triggering the relevant object (such as a user, a simulator) through the interactive interface provided by the server, or it can be sent by the relevant object through the client corresponding to the server. The instruction to obtain the sample of the image-text pair can indicate the business scenario, composition form, etc. of the image-text pair to be obtained. The business scenario to which the image-text pair to be obtained belongs can be used to limit the channels (such as Internet products that provide information flow services, XX content scenarios in related information flow services, Internet products that provide image retrieval services), methods (such as random crawling, retrieval based on retrieval information), etc. for obtaining the sample of the image-text pair. The composition form of the image-text pair to be obtained can be used to limit what kind of association relationship the image-text pair sample corresponds to, such as association relationships from the same data source, association relationships based on retrieval information, and original association relationships.

[0060] Correspondingly, the server side executes the step of acquiring the image-text pair sample according to the image-text pair sample acquisition instruction. Exemplarily, if the XX content scene indicates content in the form of pure images, the pure image can be randomly captured from the XX content scene in the relevant information flow service, and then the description information corresponding to the pure image (such as title information, text mark, etc.) is obtained, and then the image-text pair sample with the pure image as the image sample and the description information as the text sample is constructed. Among them, the image-text pair sample corresponds to the original association relationship. It is also possible to use optical character recognition (OCR) technology to perform text recognition on the pure image, and then construct an image-text pair sample with the pure image as the image sample and the text recognition result as the text sample. Among them, the image-text pair sample corresponds to the association relationship from the same data source. If the candidate image a is retrieved (crawled) based on the search term a, a picture-text pair sample with the candidate image a as the image sample and the search term a as the text sample can be constructed. Among them, the image-text pair sample corresponds to the association relationship based on the search information. It is also possible to obtain the description information corresponding to the candidate image a, and then construct an image-text pair sample with the candidate image a as the image sample and the description information as the text sample. It is also possible to perform text recognition on the candidate image a, and then construct an image-text pair sample with the candidate image a as the image sample and the text recognition result as the text sample. Thus, the image-text pair sample can be expressed in the form of a pure image or image and text.

[0061] It should be noted that A) the text recognition results can indicate text watermarks, text on emoticon images, subtitles, etc., and the meaningless text content (such as garbled characters, pure symbols (including emoticons)) therein can be filtered according to relevant data cleaning rules. B) It is also possible to randomly capture or retrieve content in the form of pictures and texts, and then obtain picture-text pair samples. C) When a video is randomly captured or retrieved, it is also possible to construct a picture-text pair sample with the video cover image as the image sample and the description information, text recognition results or search terms as the text sample. D) When searching (crawling) based on search terms, general keywords can be selected as search terms, or field keywords can be selected as search terms. In addition, the selected Internet product providing image retrieval services can be a vertical website. E) The server end that executes steps 1)-3) here and the server end that executes steps S201-S205 can be the same server end or different server ends.

[0062] In practical applications, the content published by the publishing accounts that meet the account quality requirements can be used as the data source of the image-text pair samples, and even the representative content among the published content (such as content with large exposure and content that receives more positive feedback from users) can be used as the data source of the image-text pair samples, thereby ensuring the quality of the image-text pair samples. Massive pure images can be randomly captured, and then image deduplication can be performed according to relevant data cleaning rules. In addition, image-text pair samples can also be obtained from reference data of other models with authorization to increase the amount of training data and reduce the cost of obtaining training data.

[0063] The method for acquiring image-text pair samples provided by the embodiment of the present application has high adaptability and can improve the efficiency of collecting training data. Different acquisition forms can be guided by the information carried by the image-text pair sample acquisition instruction, and these acquisition forms can be performed in parallel to quickly construct a large number of unlabeled image-text pair samples.

[0064] S302: Taking the plurality of image-text pair samples as input, using a second preset model to obtain text features corresponding to each of the plurality of text samples corresponding to the plurality of image-text pair samples;

[0065] The second preset model is a machine learning model, which can be a machine learning model in training or a machine learning model that has completed training. The second preset model can be a preset machine learning model with a Transformer structure. Multiple image-text pair samples can be input into the second preset model in sequence, and the second preset model processes each image-text pair sample in sequence, and outputs the text features corresponding to the text samples in each image-text pair sample. The second preset model realizes the extraction of text features, and the image-text pair samples are input into the second preset model. The object of the second preset model for text feature extraction is the text sample in the image-text pair sample. The text feature can indicate the result of simple vectorization processing of the text sample, or it can be a semantic feature, emotional feature or intention feature extracted from the text sample. It should be noted that step S301 and step S302 can be executed in parallel. The embodiment of the present application does not limit the execution timing and execution order of step S301 and step S302.

[0066] See also Figure 5 , taking the pure image as the image sample and the image-text pair sample constructed with the text recognition result as the text sample as an example, if the text recognition result indicates "Do you know how two cats quarrel?", the embedding input to the second preset model can be the vectorized expression of "Do you know how two cats quarrel?". Combined with the learning task in the modality missing scenario, the modality missing processing can be performed to obtain "Do you know how two [mask] [mask] [mask]?", and then the embedding corresponding to "Do you know how two [mask] [mask] [mask]?" is input into the second preset model.

[0067] S303: using the image-text feature set as input, and using a third preset model to process the image-text feature set based on the preset task information to obtain an actual processing result;

[0068] Combined with the text features and image features extracted in the aforementioned steps S301-S302, model training based on preset task information is performed here based on the image-text feature set, and the trained third preset model can be a preset machine learning model with a Transformer structure (such as an encoder in a Transformer). The multiple image-text pair samples involved in the target modality fusion model obtained in steps S301-S305 can be regarded as the current batch sample set, and accordingly, the next batch sample set including multiple image-text pair samples can be used to continue training the target modality fusion model.

[0069] Taking multiple image-text pair samples as image-text pair samples 1-10 as an example, the image-text feature set includes multiple image-text features, namely image-text features 1-10, and the image-text feature i is composed of the image features corresponding to the image sample in the image-text pair sample i and the text features corresponding to the text sample, where i is 1-10. It can be understood that each image-text pair sample carries a corresponding serial number. After the image features and text features are extracted, the image features and text features indicating the same serial number can be spliced ​​to obtain the corresponding image-text features. The image-text features 1-10 are input into the third preset model, and the correlation between the image feature part and the text feature part in the image-text feature i can be used as a "positive sample" to be learned, and the correlation between the image feature part in the image-text feature i and the text feature part in the image-text feature j can be used as a "negative sample" to be learned, where i≠j and j is 1-10. Compared with the related art in which only the correlation of one randomly sampled positive (negative) sample is learned in one iteration, the embodiment of the present application realizes the ability to learn the correlation of multiple positive samples + multiple negative samples in one iteration, which can improve the training effect and training efficiency.

[0070] In an exemplary embodiment, Figure 4 As shown, the third preset model is used to process the image-text feature set based on the preset task information to obtain an actual processing result, including:

[0071] S401: constructing a plurality of image-text feature pairs to be learned using the third preset model; wherein the plurality of image-text feature pairs to be learned are obtained by splitting and reorganizing all image-text features in the image-text feature set based on the preset task information;

[0072] S402: Determine a third correlation between the image feature part and the text feature part in each image-text feature in the image-text feature set by using the third preset model to obtain a first sub-actual processing result;

[0073] S403: using the third preset model to determine a fourth correlation between the image feature part and the text feature part in each of the image-text feature pairs to be learned, so as to obtain a second sub-actual processing result;

[0074] S404: Obtain the actual processing result based on the first sub-actual processing result and the second sub-actual processing result.

[0075] In combination with the above examples, multiple image-text feature pairs to be learned can be the splicing processing results of the image feature part in the image-text feature i and the text feature part in the image-text feature j. The third preset model is trained with multiple image-text feature pairs to be learned as negative samples and image-text feature i as positive samples. The training goal is that the third preset model can output the image-text feature pairs to be learned to indicate negative correlation between the image and the text, and the image-text feature i indicates positive correlation between the image and the text. During the training process, it is often necessary to gradually adjust the model parameters until the actual output of the model matches the target output. The embodiment of the present application involves two types of actual outputs, namely the actual output of the image-text feature pairs to be learned and the actual output of the image-text feature i. At the same time, the actual processing results involved in the embodiment of the present application are the actual outputs of all image-text feature pairs to be learned and the actual outputs of all image-text features i. Compared with guiding the adjustment of model parameters with the actual output of a certain image-text feature pair to be learned or the actual output of a certain image-text feature as the actual output result, the efficiency of model iteration can be improved, and the effect of model training can be avoided from being affected by individual deviation data.

[0076] In addition, a more fine-grained hierarchical distinction can be introduced for multiple pairs of image-text features to be learned as negative samples, so that the training data can have more fine-grained correlation information, which can improve the expression accuracy of the model. Exemplarily, the image-text feature i includes an image feature part iI and a text feature part iT, and the image-text feature pairs to be learned that are related to the image-text feature i include two categories: those that continue to use the image feature part iI (i.e., image feature part iI+text feature part jT, i≠j) and those that continue to use the text feature part iT (i.e., image feature part jI+text feature part iT, i≠j). For the former, the specific level of the negative sample (image feature part iI+text feature part jT) can be determined based on the similarity between the image feature part jl (provided by the image-text feature j that provides the text feature part jT) and the image feature part iI. For the latter, the specific level of the negative sample (image feature part jI+text feature part iT) can be determined according to the similarity between the text feature part jT (provided by the graphic feature j providing the image feature part jI) and the text feature part iT. The negative sample can include multiple levels, such as 1-5, 1 indicates the negative correlation closest to the positive correlation, and 5 indicates a completely negative correlation.

[0077] S304: adjusting the parameters of the third preset model to meet the model convergence condition according to the actual processing result and the target processing result; wherein the target processing result is determined based on the preset task information and the graphic feature set;

[0078] The target processing result indicates the target output and ideal output of the third preset model, which is the target of training. During the training process, it is often necessary to gradually adjust the model parameters until the actual output of the model matches the target output. The adjusted model parameters can be model parameters related to the learning task, and the basis for determining whether the actual output matches the target output can be a model convergence condition related to the learning task. The model convergence condition related to the learning task can define how to determine the loss value between the actual output and the target output, the loss value threshold used to determine whether the model has converged, etc.

[0079] In an exemplary embodiment, in combination with the aforementioned steps S401-S404, the target processing result includes a first sub-target processing result and a second sub-target processing result, the first sub-target processing result is used to characterize that the first correlation is positively correlated, and the second sub-target processing result is used to characterize that the second correlation is negatively correlated. Accordingly, the step of adjusting the parameters of the third preset model according to the actual processing result and the target processing result until the model convergence condition is satisfied may include the following steps: first, determining the corresponding first-class loss value based on the first sub-actual processing result and the first sub-target processing result; then, determining the corresponding second-class loss value based on the second sub-actual processing result and the second sub-target processing result; and further, adjusting the parameters of the third preset model based on the first-class loss value and the second-class loss value until the model convergence condition related to the correlation learning task is satisfied.

[0080] The multiple image-text pair samples involved in the target modal fusion model obtained in steps S301-S305 can be regarded as the current batch sample set, which provides positive samples (image-text features i) and negative samples (multiple image-text feature pairs to be learned) for the training of the third preset model. The first type of loss value is for positive samples, which indicates the similarity loss value between the third correlation of each determined image-text feature i and the positive correlation (corresponding to the first correlation). The second type of loss value is for negative samples, which indicates the similarity loss value between the fourth correlation of each determined image-text feature pair to be learned and the negative correlation (corresponding to the second correlation). The process of adjusting model parameters based on the two types of loss values ​​can be the process in which the model learns "for matching image-text pairs, close the corresponding image-text feature correlation (similarity); for unmatched image-text pairs, distance the feature correlation (similarity)". Among them, the correlation (similarity) can be represented by the vector distance between features. Through the similarity loss value, the image-text correlation is explicitly constructed to reduce the learning difficulty, so that different modal features can be mapped to the same feature space, which is beneficial to modal fusion. In addition, the correlation learning task can use the image-text matching (ImageText Match) technology.

[0081] The preset task information can also indicate other learning tasks, which are described in detail below:

[0082] 1) Learning tasks in the scenario where the preset task information indicates the missing modality:

[0083] The method of using the third preset model to process the image-text feature set based on the preset task information to obtain the actual processing result may include the following steps: first, using the third preset model to construct a plurality of image-text feature pairs with modal missing; wherein the plurality of image-text feature pairs with modal missing are obtained by performing modal missing processing on the image-text features in the image-text feature set based on the preset task information; and then, using the third preset model to determine the fifth correlation between the image feature part and the text feature part in each of the image-text feature pairs with modal missing, so as to obtain a third sub-actual processing result and use the third sub-actual processing result as the actual processing result.

[0084] Correspondingly, the target processing result is used to characterize that the first correlation is positively correlated. The adjusting the parameters of the third preset model according to the actual processing result and the target processing result until the model convergence condition is met may include the following steps: first, determining the corresponding third type of loss value based on the actual processing result and the target processing result; then, adjusting the parameters of the third preset model based on the third type of loss value until the model convergence condition related to the learning task in the modality missing scenario is met.

[0085] Exemplarily, the graphic feature i includes an image feature part iI and a text feature part iT. When constructing a graphic feature pair using the image feature part iI and the text feature part iT, the image feature part iI may be processed for modal loss, the text feature part iT may be processed for modal loss, or both the image feature part iI and the text feature part iT may be processed for modal loss. The third preset model is trained using a plurality of graphic feature pairs with modal loss as samples, and the goal of the training is that the third preset model can output graphic feature pairs with modal loss indicating that the graphic and text are positively correlated. During the training process, it is often necessary to gradually adjust the model parameters until the actual output of the model matches the target output.

[0086] Furthermore, when constructing a graphic-text feature pair using the image feature part iI and the text feature part jT, the image feature part iI may be subjected to modal loss processing, the text feature part jT may be subjected to modal loss processing, or both the image feature part iI and the text feature part jT may be subjected to modal loss processing. Where i≠j or i=j is applicable. Then, a third preset model is trained with a plurality of graphic-text feature pairs with modal loss as samples, and the training goal is that the third preset model can output a first type of graphic-text feature pair with modal loss (i.e., image feature part iI+text feature part iT) indicating positive correlation between graphic and text, and a second type of graphic-text feature pair with modal loss (i.e., image feature part iI+text feature part jT, i≠j) indicating negative correlation between graphic and text.

[0087] Through learning tasks in modality-missing scenarios, the model's associative and reasoning abilities can be improved to improve its adaptability to complex scenarios, such as processing small videos without titles. In practical applications, after obtaining multiple image-text pair samples, some of the image-text pair samples can be subjected to modality-missing processing, such as performing original modality-missing processing on 50% (of course, it can also be other percentages, such as 25%, 75%) of the image-text pair samples in the form of random padding. The original part of the filled image-text pair sample can be a partial image area of ​​an image sample, a partial text of a text sample, or both a partial image area of ​​an image sample and a partial text of a text sample. Accordingly, multiple image-text pair samples that have been subjected to modality-missing processing of some image-text pair samples are used to train the modal fusion model, that is, the "multiple image-text pair samples" involved in steps S201-S205 are replaced with "multiple image-text pair samples that have been subjected to modality-missing processing of some image-text pair samples". It should be noted that the filled and blocked parts that have been subjected to modality-missing processing do not participate in feature extraction. Modality loss processing can use methods related to Masked Language Model (MLM).

[0088] 2) The preset task information also indicates the learning task in the label prediction scenario:

[0089] The use of the third preset model to process the image-text feature set based on preset task information to obtain an actual processing result may include the following steps: using the third preset model to determine the predicted label corresponding to each image-text feature in the image-text feature set to obtain the actual processing result.

[0090] Correspondingly, the target processing result is used to indicate the label carried by the image-text pair sample. The adjusting the parameters of the third preset model according to the actual processing result and the target processing result until the model convergence condition is met may include the following steps: first, according to the correspondence between the image-text pair sample and the image-text features in the image-text feature set, the corresponding fourth category loss value is determined based on the actual processing result and the target processing result; then, based on the fourth category loss value, the parameters of the third preset model are adjusted until the model convergence condition related to the learning task in the label prediction scenario is met.

[0091] Exemplarily, taking multiple image-text pair samples as image-text pair samples 1-10 as an example, each image-text pair sample i carries a corresponding label i, and label i can be a relevant classification label of the image-text pair sample i. Classification labels can include classification labels indicating plants, classification labels indicating buildings, classification labels indicating abnormalities, etc. The image-text feature set includes multiple image-text features, namely image-text features 1-10, and image-text feature i is composed of image features corresponding to image samples in image-text pair sample i and text features corresponding to text samples, where i is 1-10. The third preset model is trained with image-text feature i as a sample, and the training goal is that the third preset model can output a predicted label for image-text feature i that matches label i. During the training process, it is often necessary to gradually adjust the model parameters until the actual output of the model matches the target output.

[0092] Through learning tasks in the label prediction scenario, the model's prediction ability can be effectively improved with the help of classification labels. The classification labels used as supervisory signals come from the acquisition of samples from images and texts, such as pure images captured or retrieved, and text labels corresponding to the images and texts, which also improves the utilization rate of relevant data.

[0093] S305: Obtain the target modality fusion model based on the first preset model, the second preset model and a third preset model that meets the model convergence condition.

[0094] See also Figure 5 The structure of the target modality fusion model includes a two-stream single modality part and a modality fusion part. The output features of the single modality part can be composed into a continuous sequence to input the modality fusion part to achieve cross-modal information interaction and fusion. The target modality fusion model can be used as a pre-training model to provide effective fusion features for downstream services. Downstream services can be content recommendation services or even content precision recommendation services. Figure 6, the pre-trained model can be fine-tuned according to the differences in tasks involved in downstream businesses. Taking the text and image content review task as an example, the pre-trained model can be used as the initial model of the text and image content review model, and the initial model can be fine-tuned using labeled data. Taking the image-only content review task as an example, part of the pre-trained model can be used as the initial model of the image-only content review model, such as the CNN model in the first preset model, and the initial model can be fine-tuned using labeled data. This can improve the modeling efficiency of information flow content processing scenarios. Furthermore, Figure 7 The iterative process that takes into account both specific tasks and pre-trained models is shown. The text and image content review model and the image-only content review model can be obtained by referring to this process.

[0095] In addition, the text samples in the image-text pair samples can also be regarded as supervisory signals, which enables the effective and sufficient training of the target modality fusion model as an image feature extraction model.

[0096] It can be seen from the technical solution provided by the above embodiment of the present application that the embodiment of the present application responds to the corpus processing instruction and determines the corresponding corpus to be processed; then, the corpus to be processed is used as input, and the corresponding fusion features are obtained by using the target modal fusion model to perform business processing based on the fusion features. The embodiment of the present application uses the target modal fusion model to process the corpus, and the fusion features outputted by it are content features that can reflect the feature relationship between modalities, which improves the reliability and effectiveness of content feature extraction, and thus improves the accuracy of content processing scenarios. The embodiment of the present application provides a model architecture for first extracting image / text features and then performing modal fusion. Each of the multiple image-text pair samples as the current batch sample set includes an image sample and a text sample; at the same time, the multiple image-text features constituting the image-text feature set correspond one-to-one to the above multiple image-text pair samples, and each image-text feature is composed of an image feature corresponding to an image sample indicating the same image-text pair sample and a text feature corresponding to a text sample. The correlation between the image feature part and the text feature part in the same graphic feature can be used as the "positive sample" to be learned, and the correlation between the image feature part and the text feature part between different graphic features can be used as the "negative sample" to be learned. The present application provides positive and negative samples for learning for the training of the target modal fusion model, ensuring the training effect. At the same time, the construction of "negative samples" no longer relies on manual work and uses a batch of sample sets as input data, which improves the training efficiency and ensures that the training obtains a target modal fusion model with high generalization ability. Furthermore, an embodiment of the present application provides a training scheme for a modal fusion model, which realizes learning from unlabeled samples based on self-supervised learning, alleviating the problem of data labeling difficulties. The obtained target modal fusion model has good adaptability in information flow content processing scenarios.

[0097] The present application also provides a corpus processing device, such as Fig. 9 As shown, the corpus processing device 90 includes:

[0098] The response module 901 is used to respond to the corpus processing instruction and determine the corresponding corpus to be processed; wherein the corpus to be processed includes at least one of the following: text to be processed and images to be processed;

[0099] Output module 902: used to take the corpus to be processed as input, obtain corresponding fusion features using the target modality fusion model, and perform business processing based on the fusion features; wherein the target modality fusion model is obtained by performing machine learning training on multiple image-text pair samples and adjusting the parameters of a preset model during the training process, each of the image-text pair samples indicates a paired image sample and a text sample, the training process includes processing an image-text feature set based on preset task information, the image-text feature set includes multiple image-text features, the multiple image-text features correspond one-to-one to the multiple image-text pair samples, the preset task information indicates a task of learning two types of correlations in the image-text feature set, the two types of correlations are respectively a first correlation between the image feature part and the text feature part in the same image-text feature in the image-text feature set, and a second correlation between the image feature part and the text feature part between different image-text features.

[0100] It should be noted that the device in the device embodiment and the method embodiment are based on the same inventive concept.

[0101] The present application embodiment also provides a content processing system 80, such as Figure 8 As shown, the following introduction Figure 8 The main functions of each service module:

[0102] 1. Content production and consumption

[0103] (1) PGC, PUGC, UGC or MCN content producers provide videos, articles, at least one image, graphics, etc. through the mobile terminal or back-end API (Application Programming Interface) system. The author can choose to actively upload the cover image of the corresponding content, which are the main sources of distributed content;

[0104] (2) Through communication with the upstream and downstream content interface services, first obtain the upload server interface address, and then upload the local file. The local file can be the result of relevant video processing after shooting. The relevant video processing can be the selection of matching music, cover image, filter template, etc. for the video, and the video can also be edited;

[0105] (3) As a consumer, it communicates with the content distribution export server to obtain the index information of the corresponding content. For videos, it communicates with the video storage server to download the corresponding streaming media files and play them through the local player; for pictures and texts, it communicates with the CDN service deployed at the edge;

[0106] (4) Report the user's browsing behavior data, reading speed, completion rate, reading time, freeze, loading time, playback clicks, etc. during the upload and download process to the server;

[0107] (5) Consumers usually browse consumption data through the feed stream. Provide consumers with a direct reporting and feedback portal related to content quality issues, and directly connect to the manual review system for confirmation and review.

[0108] 2. Uplink and Downlink Content Interface Server

[0109] (1) Communicate directly with the content production end. The content submitted from the front end is usually the title, publisher, summary, cover image, and release time of the content. If it is a shot video, you can directly access the server through the server and store the file in the video content storage service;

[0110] (2) Writing the metadata of the video content (such as file size, cover image link, bit rate, file format, title, release time, author, etc.) into the content database;

[0111] (3) Submit the uploaded files and content metadata to the dispatch center service for subsequent content processing and circulation.

[0112] 3. Content Database

[0113] (1) The core database of content. The metadata of all content published by producers is stored in this business database, with the focus on the metadata of the content itself, such as file size, cover image link, bit rate, file format, title, release time, author, video format, whether it is original or first released, and the classification of content during the manual review process (including first-, second-, and third-level classification and tag information, such as an article about XX brand mobile phones, the first-level classification is technology, the second-level classification is smart phones, the third-level classification is domestic mobile phones, and the tag information is XX brand XX model);

[0114] (2) During the manual review process, the information in the content database will be read, and the results and status of the manual review will also be sent back to the content database;

[0115] (3) The dispatch center mainly processes content through machine processing and manual review. The core of machine processing is various quality judgments (such as low-quality filtering), content classification and tag information settings, as well as content deduplication. The results will be written into the content database. Completely duplicate content will not be manually processed again, reducing the cost and investment of manual processing;

[0116] (4) When constructing a large-scale image and text pre-training model, the meta-information of the content, such as the title information of the content, will be read from the content database.

[0117] 4. Dispatch Center Services

[0118] (1) Responsible for the entire scheduling process of content flow, receiving the content through the upstream and downstream content interface servers, and then obtaining the content meta-information from the content database, such as the content title information;

[0119] (2) As the actual dispatcher and controller of the operation of the text and video links, according to the type of content, the content processing business service system is dispatched to process and review the corresponding content in the link, directly filter and mark the content with corresponding feature tags for downstream recommendation and distribution;

[0120] (3) Dispatching manual review systems and machine processing systems to control the order and priority of dispatch;

[0121] (4) After the content is enabled through a manual review system, it is directly displayed to the terminal content consumers through a content export distribution service (usually a recommendation engine or search engine or operation). In other words, the consumer side obtains the content index information, i.e., the entry address information for content access.

[0122] 5. Manual review service

[0123] (1) It is usually a web system that receives the results of machine filtering, manually confirms and reviews the results, and writes the review results into the content database for record. At the same time, the results of manual review can be used to evaluate the actual effects of machine attack and filtering models online.

[0124] (2) Report the source of the manual review task, review results, review start and end time, and other detailed review information to the statistics server;

[0125] (3) Review the data actively fed back and reported by users on the C side (consumer end), and also actively inspect the flow of online business processing. The results can be used as the data source of the business sample database, which can provide fine-tuned sample data for subsequent task iterations combined with specific content review and processing to help the model continuously upgrade, such as effectively filtering out low-quality, abnormal content suspected of "clickbait".

[0126] 6. Content storage service

[0127] (1) It is usually a group of storage servers that are widely distributed and close to users on the C side. There are usually CDN acceleration servers on the periphery for distributed cache acceleration. The content uploaded by content producers (which may include videos) is stored through the upstream and downstream content interface servers.

[0128] (2) After obtaining the content index information, the terminal consumer can also directly access the video storage server to download the corresponding content;

[0129] (3) In addition to being a data source for external services, it also serves as a data source for internal services, allowing the download file system to obtain raw video data for related processing. The channels for internal and external data sources are usually deployed separately to avoid mutual influence.

[0130] 7. Business Sample Library

[0131] (1) Save the audit results of the manual review sample as described above;

[0132] (2) Serves as a source of sample data for fine-tuning task domain models for building specific business models based on large-scale pre-trained graph and text models.

[0133] 8. Content Processing Services

[0134] (1) This means that both text and image review and video review involve a large number of quality-related models (such as models for detecting low-quality, abnormal content suspected of being "clickbait"), and the business services corresponding to these models.

[0135] 9. Image and text pre-training model

[0136] (1) According to the above training process to obtain the target modality fusion model, a large-scale image and text pre-training model is constructed. Then, combined with the business sample database, a model library at the content business level is quickly constructed, and the large-scale image and text pre-training model is used for transfer learning;

[0137] (2) Build content processing business services based on this pre-trained image and text model;

[0138] 10. Pre-training sample library

[0139] (1) Building a pre-training sample library according to the steps of obtaining image-text pair samples involved in the process of training and obtaining the target modality fusion model;

[0140] (2) Before constructing the final pre-training samples that can be used, the data is cleaned and processed accordingly to ensure the quality of the sample data.

[0141] 11. Crawler system

[0142] (1) Obtain corresponding images through various image search engines. Corresponding images usually have corresponding text titles, which are mainly used to collect pre-training sample pairs;

[0143] (2) The prototype of the crawler engine used by the crawler system here can be Puppeteer (a data package that can be used to simulate browser operation).

[0144] 12. Feedback, reporting and inspection services

[0145] (1) Collect various content quality issues actively reported and reported by consumer users, and manually review the data related to pictures and texts. The manual review results can be saved in the sample library as sample tasks for subsequent fine-tuning modeling;

[0146] (2) Active inspection of online model flow, which is manually reviewed and saved in the sample library. Inspection samples can be continuously input into the model to avoid model degradation. The model can be quickly iterated and updated on a daily basis.

[0147] It should be noted that the system and method embodiments in the system embodiment are based on the same inventive concept.

[0148] An embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the corpus processing method provided in the above method embodiment.

[0149] Further, Fig.10 The hardware structure diagram of an electronic device for implementing the corpus processing method provided in the embodiment of the present application is shown. The electronic device may participate in constituting or include the corpus processing device provided in the embodiment of the present application. Fig.10As shown, the electronic device 100 may include one or more (1002a, 1002b, ..., 1002n are used to illustrate) processors 1002 (the processor 1002 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 1004 for storing data, and a transmission device 1006 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Fig.10 The structure shown is only for illustration and does not limit the structure of the above electronic device. Fig.10 More or fewer components as shown, or with Fig.10 Different configurations are shown.

[0150] It should be noted that the one or more processors 1002 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the electronic device 100 (or mobile device). As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0151] The memory 1004 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the corpus processing method described in the embodiment of the present application. The processor 1002 executes various functional applications and data processing by running the software programs and modules stored in the memory 1004, that is, realizing the above-mentioned corpus processing method. The memory 1004 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 1004 may further include a memory remotely arranged relative to the processor 1002, and these remote memories may be connected to the electronic device 100 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0152] The transmission device 1006 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the electronic device 100. In one example, the transmission device 1006 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one embodiment, the transmission device 1006 can be a radio frequency (Radio Frequency, RF) module, which is used to communicate with the Internet wirelessly.

[0153] The display may be, for example, a touch screen liquid crystal display (LCD) that may enable a user to interact with a user interface of the electronic device 100 (or mobile device).

[0154] An embodiment of the present application also provides a computer-readable storage medium, which can be set in an electronic device to store at least one instruction or at least one program related to a corpus processing method in a method embodiment, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the corpus processing method provided by the above method embodiment.

[0155] Optionally, in this embodiment, the storage medium may be located in at least one of the multiple network servers of the computer network. Optionally, in this embodiment, the storage medium may include, but is not limited to, various media that can store program codes, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0156] It should be noted that the above-mentioned sequence of the embodiments of the present application is for description only and does not represent the advantages and disadvantages of the embodiments. The above-mentioned specific embodiments of the present application are described. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0157] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and electronic device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0158] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0159] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A corpus processing method, characterized in that: The method comprises: In response to the corpus processing instruction, the corresponding corpus to be processed is determined; wherein the corpus to be processed includes at least one of the following: graphics to be processed and images to be processed; Taking the corpus to be processed as input, the target modality fusion model is used to obtain corresponding fusion features, so as to perform business processing based on the fusion features; wherein the target modality fusion model is obtained by performing machine learning training on multiple image-text pair samples and adjusting the parameters of a preset model during the training process, each of the image-text pair samples indicates a paired image sample and a text sample, the training process includes processing an image-text feature set based on preset task information, the image-text feature set includes multiple image-text features, the multiple image-text features correspond one-to-one to the multiple image-text pair samples, the preset task information indicates a task of learning two types of correlations in the image-text feature set, the two types of correlations are respectively a first correlation between an image feature part and a text feature part in the same image-text feature in the image-text feature set, and a second correlation between an image feature part and a text feature part between different image-text features; Among them, the process of training to obtain the target modal fusion model includes: taking the multiple image-text pair samples as input, and using the first preset model to obtain the image features corresponding to each of the multiple image samples corresponding to the multiple image-text pair samples; taking the multiple image-text pair samples as input, and using the second preset model to obtain the text features corresponding to each of the multiple text samples corresponding to the multiple image-text pair samples; taking the image-text feature set as input, and using the third preset model to process the image-text feature set based on the preset task information to obtain an actual processing result; adjusting the parameters of the third preset model according to the actual processing result and the target processing result to meet the model convergence condition, and the target processing result is determined based on the preset task information and the image-text feature set; obtaining the target modal fusion model based on the first preset model, the second preset model and the third preset model that meets the model convergence condition.

2. The method according to claim 1, characterized in that The method of taking the plurality of image-text pair samples as input and using a first preset model to obtain an image feature corresponding to each of the plurality of image samples corresponding to the plurality of image-text pair samples includes: Inputting the plurality of image-text pair samples into the first preset model; At least two sub-models in the first preset model are used to process each of the multiple image samples corresponding to the multiple image-text pair samples in sequence to obtain image features corresponding to each of the image samples; wherein the at least two sub-models include a preset convolutional neural network model set at the input end and a preset machine learning model with a conversion structure set at the output end.

3. The method according to claim 1 or 2, characterized in that: The processing of the image-text feature set based on the preset task information by using the third preset model to obtain an actual processing result includes: Constructing a plurality of image-text feature pairs to be learned by using the third preset model; wherein the plurality of image-text feature pairs to be learned are obtained by splitting and reorganizing all image-text features in the image-text feature set based on the preset task information; Determine a third correlation between the image feature part and the text feature part in each image-text feature in the image-text feature set by using the third preset model to obtain a first sub-actual processing result; Determining a fourth correlation between the image feature part and the text feature part in each of the image-text feature pairs to be learned by using the third preset model, so as to obtain a second sub-actual processing result; The actual processing result is obtained based on the first sub-actual processing result and the second sub-actual processing result.

4. The method according to claim 3, characterized in that: The target processing result includes a first sub-target processing result and a second sub-target processing result, the first sub-target processing result is used to characterize that the first correlation is positively correlated, and the second sub-target processing result is used to characterize that the second correlation is negatively correlated; The adjusting the parameters of the third preset model according to the actual processing result and the target processing result to satisfy the model convergence condition includes: Determine a corresponding first-category loss value based on the first sub-actual processing result and the first sub-target processing result; Determine a corresponding second type of loss value based on the second sub-actual processing result and the second sub-target processing result; The parameters of the third preset model are adjusted based on the first type of loss value and the second type of loss value until a model convergence condition related to the relevance learning task is met.

5. The method according to claim 1, characterized in that: The preset task information further indicates a learning task in a modality-missing scenario, and the target processing result is used to characterize that the first correlation is a positive correlation; The processing of the image-text feature set based on the preset task information by using the third preset model to obtain an actual processing result includes: Constructing a plurality of image-text feature pairs with modality missing using the third preset model; wherein the plurality of image-text feature pairs with modality missing are obtained by performing modality missing processing on image-text features in the image-text feature set based on the preset task information; Determine the fifth correlation between the image feature part and the text feature part in each of the image-text feature pairs with modality loss by using the third preset model to obtain a third sub-actual processing result and use the third sub-actual processing result as the actual processing result; The adjusting the parameters of the third preset model according to the actual processing result and the target processing result to satisfy the model convergence condition includes: Determine a corresponding third-category loss value based on the actual processing result and the target processing result; The parameters of the third preset model are adjusted based on the third type of loss value until the model convergence conditions related to the learning task in the modality missing scenario are met.

6. The method according to claim 1, characterized in that: The preset task information further indicates a learning task in a label prediction scenario, and the target processing result is used to indicate a label carried by the image-text pair sample; The processing of the image-text feature set based on the preset task information by using the third preset model to obtain an actual processing result includes: Determine the predicted label corresponding to each image-text feature in the image-text feature set by using the third preset model to obtain the actual processing result; The adjusting the parameters of the third preset model according to the actual processing result and the target processing result to satisfy the model convergence condition includes: According to the correspondence between the image-text pair sample and the image-text features in the image-text feature set, a corresponding fourth type of loss value is determined based on the actual processing result and the target processing result; The parameters of the third preset model are adjusted based on the fourth type of loss value until the model convergence conditions related to the learning task in the label prediction scenario are met.

7. The method according to claim 1, characterized in that The method further comprises: In response to the image-text pair sample acquisition instruction, acquiring the image sample; Acquire text associated with the image sample; wherein the data source of the associated text is description information corresponding to the image sample, search information corresponding to the image sample, or a text recognition result of the image sample; The image-text pair sample is constructed based on the image sample and the associated text; wherein the associated text is a text sample in the image-text pair sample.

8. A corpus processing device, characterized in that: The device comprises: A response module: used to respond to the corpus processing instruction and determine the corresponding corpus to be processed; wherein the corpus to be processed includes at least one of the following: graphics and text to be processed and images to be processed; Output module: used for taking the corpus to be processed as input, obtaining corresponding fusion features by using the target modality fusion model, and performing business processing based on the fusion features; wherein the target modality fusion model is obtained by performing machine learning training on multiple image-text pair samples, and adjusting the parameters of a preset model during the training process, each of the image-text pair samples indicates a paired image sample and a text sample, the training process includes processing an image-text feature set based on preset task information, the image-text feature set includes multiple image-text features, the multiple image-text features correspond one-to-one to the multiple image-text pair samples, the preset task information indicates a task of learning two types of correlations in the image-text feature set, the two types of correlations are respectively a first correlation between an image feature part and a text feature part in the same image-text feature in the image-text feature set, and a second correlation between an image feature part and a text feature part between different image-text features; Among them, the process of training to obtain the target modal fusion model includes: taking the multiple image-text pair samples as input, and using the first preset model to obtain the image features corresponding to each of the multiple image samples corresponding to the multiple image-text pair samples; taking the multiple image-text pair samples as input, and using the second preset model to obtain the text features corresponding to each of the multiple text samples corresponding to the multiple image-text pair samples; taking the image-text feature set as input, and using the third preset model to process the image-text feature set based on the preset task information to obtain an actual processing result; adjusting the parameters of the third preset model according to the actual processing result and the target processing result to meet the model convergence condition, and the target processing result is determined based on the preset task information and the image-text feature set; obtaining the target modal fusion model based on the first preset model, the second preset model and the third preset model that meets the model convergence condition.

9. The device according to claim 8, characterized in that The method of taking the plurality of image-text pair samples as input and using a first preset model to obtain an image feature corresponding to each of the plurality of image samples corresponding to the plurality of image-text pair samples includes: Inputting the plurality of image-text pair samples into the first preset model; At least two sub-models in the first preset model are used to process each of the multiple image samples corresponding to the multiple image-text pair samples in sequence to obtain image features corresponding to each of the image samples; wherein the at least two sub-models include a preset convolutional neural network model set at the input end and a preset machine learning model with a conversion structure set at the output end.

10. The device according to claim 8 or 9, characterized in that The processing of the image-text feature set based on the preset task information by using the third preset model to obtain an actual processing result includes: Constructing a plurality of image-text feature pairs to be learned by using the third preset model; wherein the plurality of image-text feature pairs to be learned are obtained by splitting and reorganizing all image-text features in the image-text feature set based on the preset task information; Determine a third correlation between the image feature part and the text feature part in each image-text feature in the image-text feature set by using the third preset model to obtain a first sub-actual processing result; Determining a fourth correlation between the image feature part and the text feature part in each of the image-text feature pairs to be learned by using the third preset model, so as to obtain a second sub-actual processing result; The actual processing result is obtained based on the first sub-actual processing result and the second sub-actual processing result.

11. The device according to claim 10, characterized in that: The target processing result includes a first sub-target processing result and a second sub-target processing result, the first sub-target processing result is used to characterize that the first correlation is positively correlated, and the second sub-target processing result is used to characterize that the second correlation is negatively correlated; The adjusting the parameters of the third preset model according to the actual processing result and the target processing result to satisfy the model convergence condition includes: Determine a corresponding first-category loss value based on the first sub-actual processing result and the first sub-target processing result; Determine a corresponding second type of loss value based on the second sub-actual processing result and the second sub-target processing result; The parameters of the third preset model are adjusted based on the first type of loss value and the second type of loss value until a model convergence condition related to the relevance learning task is met.

12. The device according to claim 8, characterized in that: The preset task information further indicates a learning task in a modality-missing scenario, and the target processing result is used to characterize that the first correlation is a positive correlation; The processing of the image-text feature set based on the preset task information by using the third preset model to obtain an actual processing result includes: Constructing a plurality of image-text feature pairs with modality missing using the third preset model; wherein the plurality of image-text feature pairs with modality missing are obtained by performing modality missing processing on image-text features in the image-text feature set based on the preset task information; Determine the fifth correlation between the image feature part and the text feature part in each of the image-text feature pairs with modality loss by using the third preset model to obtain a third sub-actual processing result and use the third sub-actual processing result as the actual processing result; The adjusting the parameters of the third preset model according to the actual processing result and the target processing result to satisfy the model convergence condition includes: Determine a corresponding third-category loss value based on the actual processing result and the target processing result; The parameters of the third preset model are adjusted based on the third type of loss value until the model convergence conditions related to the learning task in the modality missing scenario are met.

13. The device according to claim 8, characterized in that: The preset task information further indicates a learning task in a label prediction scenario, and the target processing result is used to indicate a label carried by the image-text pair sample; The processing of the image-text feature set based on the preset task information by using the third preset model to obtain an actual processing result includes: Determine the predicted label corresponding to each image-text feature in the image-text feature set by using the third preset model to obtain the actual processing result; The adjusting the parameters of the third preset model according to the actual processing result and the target processing result to satisfy the model convergence condition includes: According to the correspondence between the image-text pair sample and the image-text features in the image-text feature set, a corresponding fourth type of loss value is determined based on the actual processing result and the target processing result; The parameters of the third preset model are adjusted based on the fourth type of loss value until the model convergence conditions related to the learning task in the label prediction scenario are met.

14. The device according to claim 8, characterized in that The device also includes a picture-text pair sample construction module; The image-text pair sample construction module is used to obtain the image sample in response to the image-text pair sample acquisition instruction; obtain the text associated with the image sample; wherein the data source of the associated text is the description information corresponding to the image sample, the search information corresponding to the image sample, or the text recognition result of the image sample; construct the image-text pair sample based on the image sample and the associated text; wherein the associated text is the text sample in the image-text pair sample.

15. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the corpus processing method according to any one of claims 1 to 7.

16. A computer-readable storage medium, characterized in that: The storage medium stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the corpus processing method according to any one of claims 1 to 7.

17. A computer program product, characterized in that The computer program product includes at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by a processor to implement the corpus processing method according to any one of claims 1 to 7.

Images