Image processing model training method, image processing method, device and equipment
By acquiring and utilizing image and text features and location features, the location information of the target object is predicted, and the image processing model is trained. This solves the problem of insufficient utilization of visual information in existing technologies and improves the model's reasoning ability and the accuracy of image processing.
Patent Information
- Application Number
- CN202111335432.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-11-11
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2041-11-11
AI Technical Summary
Existing image processing model training algorithms only focus on the matching degree between words and foreground objects, resulting in insufficient utilization of the visual information of sample image and text data, and consequently, poor reasoning ability.
By acquiring the image and text features and location features of the sample image and text data, the location information of the target object is predicted using the initial image processing model, and the model is updated to improve the inference ability of the image processing model, including feature masking and calculation of bounding box intersection-union ratio.
This improves the utilization rate of visual information in image processing models, thereby enhancing their reasoning ability and the accuracy of image processing.
Smart Images

Figure CN114118408B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a training method for an image processing model, an image processing method, a device and equipment. Background Art
[0002] At present, in order to enable electronic devices to perceive the current environment more accurately, researchers have proposed a multimodal algorithm of images and texts. They use matching image and text data pairs to train image processing models so that the image processing models can fully understand the content in the image. Thus, through this image processing model, the accuracy of image processing can be improved.
[0003] In the related art, the algorithm for training image processing models is UNITER (Universal Image-Text Representations); and the purpose of this algorithm is word region alignment, that is, to train the image processing model by matching the words in the sample text with the foreground objects in the sample image.
[0004] However, the above algorithm only focuses on the matching degree between the words in the sample text and the foreground objects in the sample image, that is, the relationship between the words and the foreground objects is determined from only one dimension, which leads to insufficient utilization of the visual information of the sample image and text data, and further leads to poor reasoning ability of the trained image processing model. Summary of the invention
[0005] The embodiments of the present disclosure provide a training method, an image processing method, an apparatus and a device for an image processing model, which can improve the reasoning ability of the image processing model. The technical solution of the present disclosure is as follows:
[0006] According to a first aspect of an embodiment of the present disclosure, a method for training an image processing model is provided, the method comprising:
[0007] Acquire sample image and text data, the sample image and text data including a sample image and a sample text, and first position information of a target object marked in the sample image and text data;
[0008] Determine the graphic features and position features of the sample graphic data, wherein the graphic features include image features of the sample image and text features of the sample text, and the position features are used to indicate the position of the target object in the sample graphic data;
[0009] Based on the graphic feature and the position feature, predicting the second position information of the target object through an initial image processing model;
[0010] Based on the first position information and the second position information, the initial image processing model is updated to obtain an image processing model.
[0011] In a possible implementation, the target object includes at least one of a target object in the sample image and target text information in the sample text, and the process of determining the position feature of the sample image and text data includes:
[0012] Determine at least one of a first position feature of the sample image and a second position feature of the sample text, wherein the first position feature is used to indicate the position of the target object in the sample image, and the second position feature is used to indicate the position of the target text information in the sample text;
[0013] At least one of the first position feature and the second position feature is determined as the position feature of the sample graphic data.
[0014] In another possible implementation, predicting the second position information of the target object by using an initial image processing model based on the graphic feature and the position feature includes:
[0015] When the position feature includes the first position feature, superimposing the image feature and the first position feature, and predicting the second position information of the target object through the initial image processing model based on the superimposed image feature and the text feature;
[0016] When the position feature includes the second position feature, superimposing the text feature and the second position feature, and predicting the second position information of the target text information through the initial image processing model based on the superimposed text feature and the image feature;
[0017] When the position feature includes the first position feature and the second position feature, the image feature and the first position feature are superimposed, and the text feature and the second position feature are superimposed. Based on the superimposed text features and the superimposed image features, the second position information of the target object and the second position information of the target text information are predicted through the initial image processing model.
[0018] In another possible implementation, there are multiple target objects, and the graphic features include multiple graphic features corresponding to the multiple target objects; the second position information is the position of the target object in the sample graphic data;
[0019] The predicting the second position information of the target object by using an initial image processing model based on the graphic feature and the position feature includes:
[0020] Performing feature masking processing on a first graphic feature among the plurality of graphic features;
[0021] Based on the position feature and the remaining graphic features, the position of the target object corresponding to the first graphic feature that is covered is predicted through the initial image processing model.
[0022] In another possible implementation, there are multiple target objects, the graphic features include multiple graphic features corresponding to the multiple target objects, and the second position information is the position relationship between any two target objects in the sample graphic data;
[0023] The predicting the second position information of the target object by using an initial image processing model based on the graphic feature and the position feature includes:
[0024] Obtaining bounding boxes corresponding to any two second graphic features, where the bounding boxes are bounding boxes of target objects corresponding to the second graphic features;
[0025] Determine an intersection-over-union ratio of the two bounding boxes, where the intersection-over-union ratio is a ratio of the intersection to the union of the two bounding boxes;
[0026] Based on the intersection-over-union ratio, the positional relationship of the target objects corresponding to the two bounding boxes is predicted through the initial image processing model.
[0027] In another possible implementation manner, the performing feature masking processing on the first graphic feature among the graphic features includes at least one of the following implementation manners:
[0028] Performing feature masking processing on the image features in the first graphic features; or,
[0029] Perform feature masking processing on the text features in the first graphic features.
[0030] In another possible implementation manner, there are multiple target objects, and the location features include multiple location features corresponding to the multiple target objects;
[0031] The predicting the second position information of the target object by using an initial image processing model based on the graphic feature and the position feature includes:
[0032] Performing feature mask processing on a first position feature among the plurality of position features;
[0033] Based on the graphic features and the remaining position features, the initial image processing model is used to predict the second position information of the target object corresponding to the first position feature that is covered.
[0034] In another possible implementation, predicting the second position information of the target object by using an initial image processing model based on the graphic feature and the position feature includes:
[0035] Determine a first sample feature based on the image-text feature, the position feature and at least one mark feature, wherein the at least one mark feature is used to identify a positional relationship between the image feature and the text feature;
[0036] Based on the sample features, second position information of the target object is predicted by the initial image processing model.
[0037] In another possible implementation, determining the first sample feature based on the graphic feature, the position feature, and at least one marker feature includes:
[0038] Fusing the image feature, the text feature and the position feature to determine a second sample feature;
[0039] Inserting the first signature feature, the second signature feature and the third signature feature into the second sample feature respectively to obtain the first sample feature;
[0040] The first signature feature is between the image features, and the first signature feature is a global feature of the sample image;
[0041] The second signature feature is between the text features, and the second signature feature is a global feature of the sample text;
[0042] The third marker feature is located at the end of the second sample feature, and the third marker feature is a terminator.
[0043] In another possible implementation, the method further includes:
[0044] Get test sample data;
[0045] Based on the test sample data, the model parameters of the image processing model are adjusted.
[0046] According to a second aspect of an embodiment of the present disclosure, there is provided an image processing method, the method comprising:
[0047] Acquire target graphic data, wherein the target graphic data includes a target image and a target text, and the target text includes question information;
[0048] Based on the image processing model, the target image and the target text are processed to obtain an image processing result. The image processing result is answer information corresponding to the question information determined based on the target image. The target image processing model is trained based on the image and text features and position features of sample image and text data.
[0049] In a possible implementation, the processing of the target image and the target text based on the image processing model to obtain an image processing result includes:
[0050] Based on the image processing model, determining a target image feature of the target image, a target text feature of the target text, and a target position feature, wherein the target position feature is used to represent a position of a target object in the target image and the target text;
[0051] An image processing result is determined based on the target image feature, the target text feature and the target position feature.
[0052] In another possible implementation, the target object includes at least one of a target object in the target image and target text information in the target text, and the process of determining the target position feature includes:
[0053] Determine at least one of a third position feature of the target image and a fourth position feature of the target text, wherein the third position feature is used to indicate the position of the target object in the target image, and the fourth position feature is used to indicate the position of the target text information in the target text;
[0054] At least one of the third position feature and the fourth position feature is determined as the target position feature.
[0055] In another possible implementation, determining the image processing result based on the target image feature, the target text feature, and the target position feature includes:
[0056] Masking is performed on some of the target image features and the target text features.
[0057] Based on the remaining features, predicting the position information corresponding to the masked features through the image processing model;
[0058] An image processing result is determined based on the position information corresponding to the masked feature.
[0059] In another possible implementation manner, the performing mask processing on some features of the target image features and the target text features based on the image processing model includes at least one of the following implementation manners:
[0060] Performing feature masking processing on the target image features; or,
[0061] Perform feature masking processing on the target text features.
[0062] In another possible implementation manner, the number of the target objects is multiple, and the target location feature includes multiple target location features corresponding to the multiple target objects;
[0063] The determining of the image processing result based on the target image feature, the target text feature and the target position feature includes:
[0064] Performing feature mask processing on the third position feature among the multiple target position features;
[0065] Based on the remaining features, predicting the position information corresponding to the masked third position feature through the image processing model;
[0066] An image processing result is determined based on the position information corresponding to the masked third position feature.
[0067] In another possible implementation, there are multiple target objects, and determining the image processing result based on the target image feature, the target text feature, and the target position feature includes:
[0068] Get the bounding boxes corresponding to any two target objects;
[0069] Determine an intersection-over-union ratio of the two bounding boxes, where the intersection-over-union ratio is a ratio of the intersection to the union of the two bounding boxes;
[0070] Based on the intersection-over-union ratio, predicting the positional relationship between the two target objects through the image processing model;
[0071] An image processing result is determined based on the positional relationship between the two target objects.
[0072] According to a third aspect of an embodiment of the present disclosure, a training device for an image processing model is provided, the device comprising:
[0073] A first acquisition module is used to acquire sample graphic data, the sample graphic data including sample images and sample texts, and first position information of a target object marked in the sample graphic data;
[0074] A determination module, used to determine the graphic features and position features of the sample graphic data, wherein the graphic features include the image features of the sample image and the text features of the sample text, and the position features are used to indicate the position of the target object in the sample graphic data;
[0075] A prediction module, configured to predict the second position information of the target object through an initial image processing model based on the image and text features and the position features;
[0076] An updating module is used to update the initial image processing model based on the first position information and the second position information to obtain an image processing model.
[0077] In a possible implementation, the determining module includes:
[0078] A first determining unit, configured to determine at least one of a first position feature of the sample image and a second position feature of the sample text, wherein the first position feature is used to indicate the position of the target object in the sample image, and the second position feature is used to indicate the position of the target text information in the sample text;
[0079] The second determining unit is used to determine at least one of the first position feature and the second position feature as the position feature of the sample graphic data.
[0080] In another possible implementation, the prediction module includes:
[0081] A first prediction unit is configured to, when the position feature includes the first position feature, superimpose the image feature and the first position feature, and predict second position information of the target object through the initial image processing model based on the superimposed image feature and the text feature;
[0082] A second prediction unit is used for, when the position feature includes the second position feature, superimposing the text feature and the second position feature, and predicting the second position information of the target text information through the initial image processing model based on the text feature and the image feature obtained by superimposing;
[0083] A third prediction unit is used for, when the position feature includes the first position feature and the second position feature, superimposing the image feature and the first position feature, superimposing the text feature and the second position feature, and predicting the second position information of the target object and the second position information of the target text information through the initial image processing model based on the superimposed text features and the superimposed image features.
[0084] In another possible implementation, the prediction module includes:
[0085] A first processing unit, configured to perform feature masking processing on a first image and text feature among the plurality of image and text features;
[0086] The fourth prediction unit is used to predict the position of the target object corresponding to the first image and text feature that is covered through the initial image processing model based on the position feature and the remaining image and text features.
[0087] In another possible implementation, the prediction module includes:
[0088] An acquisition unit, configured to acquire a bounding box corresponding to any two second graphic features, wherein the bounding box is a bounding box of a target object corresponding to the second graphic features;
[0089] a third determining unit, configured to determine an intersection-and-union ratio of the two bounding boxes, wherein the intersection-and-union ratio is a ratio of the intersection to the union of the two bounding boxes;
[0090] A fifth prediction unit is used to predict the positional relationship of the target objects corresponding to the two bounding boxes based on the intersection-over-union ratio and through the initial image processing model.
[0091] In another possible implementation, the first processing unit is configured to perform feature masking processing on image features in the first graphic features; or perform feature masking processing on text features in the first graphic features.
[0092] In another possible implementation, the prediction module includes:
[0093] A second processing unit, configured to perform feature mask processing on a first position feature among the plurality of position features;
[0094] The sixth prediction unit is used to predict the second position information of the target object corresponding to the first position feature that is covered based on the graphic feature and the remaining position features through the initial image processing model.
[0095] In another possible implementation, the prediction module includes:
[0096] a fourth determining unit, configured to determine a first sample feature based on the image-text feature, the position feature, and at least one mark feature, wherein the at least one mark feature is used to identify a positional relationship between the image feature and the text feature;
[0097] A seventh prediction unit is used to predict second position information of the target object through the initial image processing model based on the first sample feature.
[0098] In another possible implementation, the fourth determination unit is used to fuse the image feature, the text feature and the position feature to determine the second sample feature; insert the first marker feature, the second marker feature and the third marker feature into the second sample feature respectively to obtain the first sample feature; the first marker feature is before the image feature, and the first marker feature is a global feature of the sample image; the second marker feature is between the text features, and the second marker feature is a global feature of the sample text; the third marker feature is located at the end of the second sample feature, and the third marker feature is a terminator.
[0099] In another possible implementation, the device further includes:
[0100] A second acquisition module is used to acquire test sample data;
[0101] An adjustment module is used to adjust the model parameters of the image processing model based on the test sample data.
[0102] According to a fourth aspect of an embodiment of the present disclosure, there is provided an image processing device, the device comprising:
[0103] An acquisition module, used for acquiring target graphic data, wherein the target graphic data includes a target image and a target text, and the target text includes question information;
[0104] A processing module is used to process the target image and the target text based on an image processing model to obtain an image processing result, wherein the image processing result is answer information corresponding to the question information determined based on the target image, and the target image processing model is trained based on the image and text features and position features of sample image and text data.
[0105] In a possible implementation, the processing module includes:
[0106] A first determining unit is used to determine a target image feature of the target image, a target text feature of the target text, and a target position feature based on the image processing model, wherein the target position feature is used to indicate a position of a target object in the target image and the target text;
[0107] The second determining unit is used to determine the image processing result based on the target image feature, the target text feature and the target position feature.
[0108] In another possible implementation, the first determination unit is used to determine at least one of a third position feature of the target image and a fourth position feature of the target text, wherein the third position feature is used to indicate the position of the target object in the target image, and the fourth position feature is used to indicate the position of the target text information in the target text; and determine at least one of the third position feature and the fourth position feature as the target position feature.
[0109] In another possible implementation, the second determination unit is used to mask some features of the target image features and the target text features, and based on the remaining features, predict position information corresponding to the masked features through the image processing model; and determine the image processing result based on the position information corresponding to the masked features.
[0110] In another possible implementation manner, the second determining unit is configured to perform feature masking processing on the target image features; or perform feature masking processing on the target text features.
[0111] In another possible implementation, the second determination unit is used to perform feature masking processing on the third position feature among the multiple target position features; based on the remaining features, predict the position information corresponding to the masked third position feature through the image processing model; and determine the image processing result based on the position information corresponding to the masked third position feature.
[0112] In another possible implementation, the second determination unit is used to obtain bounding boxes corresponding to any two target objects; determine an intersection-and-union ratio of the two bounding boxes, where the intersection-and-union ratio is a ratio of the intersection to the union of the two bounding boxes; based on the intersection-and-union ratio, predict the positional relationship between the two target objects through the image processing model; and determine an image processing result based on the positional relationship between the two target objects.
[0113] According to a fifth aspect of an embodiment of the present disclosure, an electronic device is provided, comprising a processor and a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement a training method for an image processing model or an image processing method in the above-mentioned embodiment.
[0114] According to the sixth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of a terminal, the terminal is enabled to execute the training method of the image processing model or the image processing method in the above-mentioned embodiment.
[0115] According to a seventh aspect of an embodiment of the present disclosure, a computer program product is provided, wherein the computer program product comprises a computer program, and when the computer program is executed by a processor, the training method of the image processing model or the image processing method in the above-mentioned embodiment is implemented.
[0116] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:
[0117] In the disclosed embodiments, the position information of the target object is predicted by the graphic features and position features of the sample graphic data, thereby training the image processing model so that the image processing model can make full use of the position features of the sample graphic data, improve the utilization rate of the visual information in the image, and thus improve the reasoning ability of the image processing model.
[0118] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0119] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.
[0120] Figure 1 is a schematic diagram of an implementation environment of a training method for an image processing model according to an exemplary embodiment;
[0121] Figure 2 is a flowchart of a method for training an image processing model according to an exemplary embodiment;
[0122] Figure 3 is a flowchart of an image processing method according to an exemplary embodiment;
[0123] Figure 4 is a flowchart of a method for training an image processing model according to an exemplary embodiment;
[0124] Figure 5 is a schematic diagram of a training image processing model according to an exemplary embodiment;
[0125] Figure 6 is a flowchart of a method for training an image processing model according to an exemplary embodiment;
[0126] Figure 7 is a flowchart of a method for training an image processing model according to an exemplary embodiment;
[0127] Figure 8 is a flowchart of an image processing method according to an exemplary embodiment;
[0128] Fig. 9 is a flowchart of an image processing method according to an exemplary embodiment;
[0129] Fig.10 is a block diagram of a training device for an image processing model according to an exemplary embodiment;
[0130] Fig.11 is a block diagram of an image processing device according to an exemplary embodiment;
[0131] Fig.12 is a block diagram of a terminal according to an exemplary embodiment;
[0132] Fig.13 It is a block diagram of a server according to an exemplary embodiment. DETAILED DESCRIPTION
[0133] In order to make the purpose, technical solution and advantages of the present disclosure clearer, the embodiments of the present disclosure will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.
[0134] It is understood that the terms "first", "second", etc. used in the present disclosure may be used herein to describe various concepts, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the application, the first position information may be referred to as the second position information, and similarly, the second position information may be referred to as the first position information.
[0135] The terms "at least one", "multiple", "each" and "any" used in the present disclosure, at least one includes one, two or more than two, multiple includes two or more than two, each refers to each of the corresponding multiple, and any refers to any one of the multiple. For example, the multiple graphic features include 3 graphic features, and each refers to each of the 3 graphic features. Any refers to any one of the 3 graphic features, which can be the first, the second or the third.
[0136] Figure 1 is a schematic diagram of an implementation environment of a training method for an image processing model according to an exemplary embodiment. Figure 1 As shown, the implementation environment includes a terminal 101 and a server 102 .
[0137] Optionally, the terminal 101 is a smart phone, a tablet computer, a laptop computer, a desktop computer, etc., but is not limited thereto. The terminal 101 may generally refer to one of a plurality of terminals, and the present embodiment is only illustrated by the terminal 101. Those skilled in the art may know that the number of the above terminals may be more or less. The terminal 101 may run various types of applications, such as video applications, image applications, etc. The terminal 101 may be connected to the server 102 via a wireless network or a wired network, and may upload data to the server 102 via a wireless network or a wired network, and the data may be graphic data collected by the terminal 101.
[0138] Optionally, server 102 is an independent physical server, or a server cluster or distributed file system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.
[0139] The server 102 is used to execute a training method of an image processing model provided in an embodiment of the present disclosure. The server 102 and the terminal 101 can be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiment of the present disclosure. Optionally, the number of the above-mentioned servers 102 can be more or less, which is not limited in the embodiment of the present disclosure. Of course, the server 102 can also include other functional servers to provide more comprehensive and diversified services.
[0140] It should be noted that the graphic data processed by the embodiment of the present disclosure can be uploaded to the server by the terminal, and can also be obtained by the server itself, and the embodiment of the present disclosure does not limit this.
[0141] Another point that needs to be explained is that, in some embodiments, the server processes graphic data and trains the image processing model. In other embodiments, the terminal can also train the image processing model locally. In the following embodiments, the server training the image processing model is used as an example for explanation.
[0142] After introducing the implementation environment of the embodiment of the present disclosure, the application scenario of the embodiment of the present disclosure will be introduced in combination with the above implementation environment. It should be noted that in the following description, the terminal is the above terminal 101, and the server is the above server 102.
[0143] Through a training method for an image processing model provided by an embodiment of the present disclosure, after the terminal uploads graphic data to the server, the server trains the image processing model based on the graphic data. The image processing model can be applied to downstream tasks such as visual question answering, visual common sense reasoning or image retrieval. Of course, with the development of science and technology, the training method for the image processing model provided by the embodiment of the present disclosure can also be applied to other scenarios for model training, and the embodiment of the present disclosure does not limit this.
[0144] The training method of the image processing model provided by the embodiments of the present disclosure can be applied to any scenario based on image reasoning information.
[0145] In the first scenario, the training method of the image processing model can be applied to the scenario of visual question answering. For example, a visually impaired user who is not clear about the color of the flower in the image can input the image and a question about the color of the flower in the image into the terminal, and the terminal sends the image and the question to the server. The server calls the trained image processing model, determines the answer to the question through the model, outputs the answer, and feeds it back to the user, so that the user can know the color of the flower in the image.
[0146] In the second scenario, the training method of the image processing model can be applied to visual common sense reasoning scenarios. For example, if a traffic management personnel needs to know whether there are vehicles violating traffic rules in the surveillance image, they can input the surveillance image and questions about the vehicles violating traffic rules in the image into the terminal. The terminal sends the surveillance image and questions to the server, and the server calls the trained image processing model, infers the vehicles violating traffic rules and the reasons through the model, and feeds back the vehicles violating traffic rules and the reasons to the traffic management personnel.
[0147] After introducing the implementation environment and application scenarios of the embodiments of the present disclosure, the training method of the image processing model provided by the embodiments of the present disclosure is described below. Figure 2 1 is a flowchart of a method for training an image processing model according to an exemplary embodiment. The disclosed embodiment is executed by a server, and the method includes the following steps:
[0148] Step 201: The server obtains sample image and text data, where the sample image and text data includes a sample image and a sample text, and first position information of a target object marked in the sample image and text data.
[0149] Step 202: The server determines the graphic features and position features of the sample graphic data. The graphic features include the image features of the sample image and the text features of the sample text. The position features are used to indicate the position of the target object in the sample graphic data.
[0150] Step 203: The server predicts the second position information of the target object through an initial image processing model based on the image and text features and the position features.
[0151] Step 204: The server updates the initial image processing model based on the first position information and the second position information to obtain an image processing model.
[0152] In the disclosed embodiment, the server predicts the location information of the target object through the graphic features and position features of the sample graphic data, and trains the image processing model from multiple dimensions, so that the image processing model can make full use of the feature information of the sample graphic data, improve the utilization rate of the visual information in the image, and thus improve the reasoning ability of the image processing model.
[0153] Figure 3 1 is a flowchart of an image processing method according to an exemplary embodiment. The disclosed embodiment is executed by a server, and the method includes the following steps:
[0154] Step 301: The server obtains target graphic data, the target graphic data includes a target image and a target text, and the target text includes question information.
[0155] Step 302: The server processes the target image and target text based on the image processing model to obtain an image processing result. The image processing result is answer information corresponding to the question information determined based on the target image. The target image processing model is trained based on the image and text features and position features of the sample image and text data.
[0156] In the disclosed embodiment, the server calls a trained image processing model to process the target image and text data. Since the image processing model is trained through the image and text features and position features of the sample image and text data, it has strong reasoning ability and can make full use of the feature information of the target image and text data. Therefore, the image processing result obtained by the image processing model is more accurate, and this method improves the accuracy of image processing.
[0157] Figure 4 1 is a flowchart of a method for training an image processing model according to an exemplary embodiment. The disclosed embodiment is executed by a server, and the method includes the following steps:
[0158] Step 401: The server obtains sample graphic and text data.
[0159] The sample image and text data includes a sample image and a sample text. The first position information of the target object is marked in the sample image and text data. The sample image and the sample text match each other, that is, the sample text can represent the content information in the sample image. Before training the image processing model, the server needs to determine the sample image and text data first, so that the image processing model can be trained based on the sample image and text data in the subsequent process.
[0160] In one possible implementation, the sample image and text data is generated by a server. Accordingly, the steps for the server to obtain the sample image and text data are as follows: the server obtains multiple sample images and multiple sample texts, matches the multiple sample images and the multiple sample texts one by one, determines multiple pairs of sample images and sample texts with high matching degrees, and determines the sample image and text data based on the multiple pairs of matching sample images and sample texts.
[0161] In the disclosed embodiment, the server itself generates sample graphic data and can purposefully acquire the sample graphic data, that is, it can acquire graphic data that meets the requirements as samples, and the model trained with the sample graphic data better meets the requirements.
[0162] In a possible implementation, the sample image and text data comes from a constructed data set, which contains multiple pairs of matching sample images and sample texts. The server directly calls the constructed data set to obtain the sample image and text data. Accordingly, the step for the server to obtain the sample image and text data is: the server obtains the constructed data set, and obtains the sample image and text data from the data set. For example, the data set is a caption photo data set, a COCO (Common Objects in Context) data set, or a CC (Conceptual Captions) data set, etc. In the embodiments of the present disclosure, the form of the data set is not specifically limited.
[0163] In the disclosed embodiment, the server directly calls the constructed data set to obtain sample graphic data, which is simple to operate and has low running consumption.
[0164] One point that needs to be explained is that, in the process of training the image processing model, the server can also obtain multiple negative samples, match the sample image and text data, and train the image processing model, that is, for each sample image and text data, the server selects multiple negative samples to match the sample image and text data, so as to train the image processing model. Among them, the negative sample is a sample image or sample text that does not match the sample image and text data, that is, the content information of the negative sample is irrelevant to the content information of the sample image and text data. For example, the negative sample is a sample text that does not match the sample image in the sample image and text data, and the server selects multiple negative samples to match the sample image in the sample image and text data, so as to train the image processing model. For example, the negative sample is a sample image that does not match the sample text in the sample image and text data, and the server selects multiple negative samples to match the sample text in the sample image and text data, so as to train the image processing model. The server can select 15 negative samples to match the sample image and text data, which is not specifically limited here.
[0165] Step 402: The server determines the graphic features of the sample graphic data.
[0166] The graphic features include the image features of the sample image and the text features of the sample text. Accordingly, the process of the server determining the graphic features of the sample graphic data can be implemented through steps 4021-4022, including:
[0167] Step 4021: The server determines image features of the sample image.
[0168] Before determining the image features of the sample image, the server obtains multiple images and trains the image feature model to obtain a trained image feature model. Afterwards, the server determines the image features of the sample image using the trained image feature model. For example, see Figure 5 , the server inputs the sample image into the trained image feature model, and determines the image features of the sample image through the image feature model. In the disclosed embodiment, the image feature model may be Faster R-CNN (Faster Region-CNN) or CNN (Convolutional Neural Network), etc., which are not specifically limited here. The server inputs the sample image into the trained Faster R-CNN network, and the Faster R-CNN network determines the image features of the sample image. The image features may be features of the target object in the sample image, and each sample image corresponds to multiple image features.
[0169] Step 4022: The server determines the text features of the sample text.
[0170] Before determining the text features of the sample text, the server obtains multiple texts and trains the image feature model to obtain a trained text feature model. Afterwards, the server uses the text features of the sample text of the trained text feature model, for example, Figure 5 , the server inputs the sample text into a trained text feature model, and determines the text features of the sample text through the text feature model. For example, the text feature model is a BERT (Bidirectional Encoder Representations from Transformer) or a RoBERta (Robustly Optimized BERT Pretraining Approach) model, etc., which is not specifically limited here. For example, the server inputs the sample text into a trained RoBERta model, and the RoBERta model determines the text features of the sample text. The text features can be features of the target text information in the sample text, and each sample text corresponds to multiple text features.
[0171] Step 403: The server determines the location characteristics of the sample graphic data.
[0172] The position feature is used to indicate the position of the target object in the sample image and text data. The target object includes at least one of the target object in the sample image and the target text information in the sample text. Accordingly, the process of determining the position feature of the sample image and text data can be implemented by the following steps 4031-4032, including:
[0173] Step 4031: The server determines at least one of a first position feature of the sample image and a second position feature of the sample text.
[0174] The first position feature is used to indicate the position of the target object in the sample image, and the second position feature is used to indicate the position of the target text information in the sample text. Figure 5 , the server encodes the position of the target object in the sample image to obtain a first position feature, and encodes the position of the target text information in the sample text to obtain a second position feature. For example, the server uses a one-hot (one-bit effective encoding) encoding method to encode the position of the target object in the sample image and the position of the target text information in the sample text, respectively, to determine the first position feature and the second position feature. In the embodiments disclosed herein, the encoding method is not specifically limited.
[0175] Step 4032: The server determines at least one of the first location feature and the second location feature as a location feature of the sample graphic data.
[0176] Correspondingly, the server determines the first position feature as the position feature of the sample graphic data, or the server determines the second position feature as the position feature of the sample graphic data, or the server determines the first position feature and the second position feature as the position features of the sample graphic data.
[0177] In the disclosed embodiment, the server obtains at least one of the position features of the target object and the position features of the target text information for use in training the image processing model in a subsequent process, thereby adding an information dimension to training the model and improving the utilization rate of the visual information in the image, thereby improving the reasoning ability of the image processing model.
[0178] Step 404: The server determines a third sample feature based on the image and text features and the location features.
[0179] The third sample feature is a feature obtained by superimposing the image and text features and the location features. The server superimposes the image and text features and the location features according to the different location features to determine the third sample feature. The process can be divided into three cases, including:
[0180] In the first case, when the location feature includes the first location feature, the server superimposes the graphic feature and the location feature, and the process of determining the third sample feature is as follows: the server superimposes the image feature and the first location feature to determine the third sample feature, that is, the server adds the feature vector of the image feature and the feature vector of the first location feature to obtain the third sample feature.
[0181] In the second case, when the position feature includes the second position feature, the server superimposes the graphic feature and the position feature, and the process of determining the third sample feature is: the server superimposes the text feature and the second position feature to determine the third sample feature, that is, the server adds the feature vector of the text feature and the feature vector of the second position feature to obtain the third sample feature.
[0182] In the third case, when the position feature includes the first position feature and the second position feature, the server superimposes the graphic feature and the position feature, and the process of determining the third sample feature is as follows: the server superimposes the image feature and the first position feature, and superimposes the text feature and the second position feature, and determines the third sample feature, that is, the server adds the feature vector of the image feature and the feature vector of the first position feature, and adds the feature vector of the text feature and the feature vector of the second position feature to obtain the third sample feature.
[0183] Step 405: The server predicts the second position information of the target object through the initial image processing model based on the third sample feature.
[0184] In a possible implementation, the server directly uses the third sample feature to predict the second position information of the target object through the initial image processing model. Accordingly, the process of the server predicting the second position information of the target object is: the server predicts the second position information of the target object through the initial image processing model based on the superimposed image features (third sample features) and text features, or the server predicts the second position information of the target text information through the initial image processing model based on the superimposed text features (third sample features) and image features, or the server predicts the second position information of the target object and the second position information of the target text information through the initial image processing model based on the superimposed text features and the superimposed image features (third sample features).
[0185] In the disclosed embodiment, the server predicts the location information of the target object by superimposing the graphic features and position features, and trains the image processing model from multiple dimensions, so that the image processing model can make full use of the feature information of the sample graphic data, improve the utilization rate of the visual information in the image, and thus improve the reasoning ability of the image processing model.
[0186] In another possible implementation, the server may also insert a marker feature into the third sample feature, and then predict the second position information of the target object through the initial image processing model. In the subsequent process of predicting the second position information of the target object through the initial image processing model, the marker feature prompts the initial image processing model of the position of the graphic features and the position features. Accordingly, the step of the server predicting the second position information of the target object is: the server determines the first sample feature based on the graphic features, the position features and at least one marker feature, and the at least one marker feature is used to identify the positional relationship between the image features and the text features; based on the first sample feature, the second position information of the target object is predicted through the initial image processing model.
[0187] In the disclosed embodiment, the server predicts the location information of the target object by inserting the graphic features and position features of the mark, and trains the image processing model from multiple dimensions, so that the image processing model can fully utilize the feature information of the sample graphic data, improve the utilization rate of the visual information in the image, and thus improve the reasoning ability of the image processing model.
[0188] Before inserting the marker feature, the server needs to fuse the graphic feature and the position feature. Accordingly, the server determines the first sample feature based on the graphic feature, the position feature and at least one marker feature through the following two steps, including:
[0189] (1) The server integrates the image features, text features, and location features to determine the second sample features.
[0190] The server superimposes the image feature with the position feature corresponding to the image feature (first position feature) to obtain a superimposed image feature, superimposes the text feature with the position feature corresponding to the text feature (second position feature) to obtain a superimposed text feature, splices the superimposed image feature and the superimposed text feature to obtain a second sample feature, and then splices the obtained third sample feature to obtain a second sample feature.
[0191] (2) The server inserts the first signature feature, the second signature feature, and the third signature feature into the second sample feature respectively to obtain the first sample feature.
[0192] For example, see Figure 5 , wherein the first marker feature is before the image feature, and the first marker feature is a global feature of the sample image; the second marker feature is before the text feature, and the second marker feature is a global feature of the sample text; the third marker feature is located at the end of the second sample feature, and the third marker feature is a terminator. When the server calls the image processing model to process the image and text data, the first marker feature is used to prompt the image processing model to start the image data, the second marker feature is used to prompt the image processing model to start the text data, and the third marker feature is used to prompt the image processing model to end the image and text data.
[0193] For example, the first marker feature is "[IMG]", the second marker feature is "[CLS]", and the third marker feature is "[SEP]", then the first sample feature obtained by the server is "[IMG]+superimposed image feature+[CLS]+superimposed text feature+[SEP]", or the first sample feature obtained is "[CLS]+superimposed text feature+[IMG]+superimposed image feature+[SEP]".
[0194] In the disclosed embodiment, the server inserts a marker feature into the superimposed graphic features and position features, which can accurately identify the positions of the graphic features and position features in the superimposed features, facilitate subsequent prediction of the target object's position information, and improve the accuracy of the prediction.
[0195] Step 406: The server updates the initial image processing model based on the first position information and the second position information to obtain an image processing model.
[0196] Among them, the first position information is the position information of the target object marked in the sample image and text data, and the second position information is the position information of the target object predicted by the initial image processing model. The server compares the first position information with the second position information, updates the model parameters of the initial image processing model, and performs multiple training to make the second position information predicted by the model as close as possible to the first position information, thereby obtaining an image processing model. Compared with the initial image processing model, the prediction performance of this model is greatly improved.
[0197] One point that needs to be explained is that after the training is completed, the server can also test the trained image processing model, adjust the model parameters according to the test results, and further optimize the image processing model. Correspondingly, the steps for the server to optimize the image processing model are: the server obtains test sample data; based on the test sample data, the model parameters of the image processing model are adjusted. The test sample data can be a graphic data set for downstream tasks. For example, the downstream task is VCR (Visual Commonsense Reasoning) or VQA (Visual Question Answering), which is not specifically limited here. The server uses the graphic data set of the downstream task to test the model, adjust the model parameters, and improve the accuracy of the model in performing downstream tasks.
[0198] In the disclosed embodiment, the server fuses the graphic features and position features of the sample graphic data, predicts the position of the target object in the sample graphic data, and trains the image processing model from multiple dimensions, so that the image processing model can make full use of the feature information of the sample graphic data, improve the utilization rate of the visual information in the image, and thus improve the reasoning ability of the image processing model.
[0199] Figure 6 1 is a flowchart of a method for training an image processing model according to an exemplary embodiment. The disclosed embodiment is executed by a server, and the method includes the following steps:
[0200] Step 601: The server obtains sample graphic and text data.
[0201] Step 602: The server determines the graphic features of the sample graphic data.
[0202] Step 603: The server determines the location characteristics of the sample graphic data.
[0203] Steps 601-603 are the same as steps 401-403 and will not be repeated here.
[0204] Step 604: The server performs feature masking based on the image and text features and the position features.
[0205] Depending on the features, the server's feature masking process can be divided into the following cases:
[0206] In the first case, the server masks the graphic features of the target object through the initial image processing model. Accordingly, the server performs feature masking processing based on the graphic features and position features as follows: the server performs feature masking processing on the first graphic feature among multiple graphic features.
[0207] Since there are multiple target objects in the sample graphic data, the graphic features include multiple graphic features corresponding to the multiple target objects, wherein the first graphic feature is the graphic feature corresponding to the electronically selected target object. The server can randomly select one or more target objects, mask the first graphic feature of the target object, and thus predict the position of the target object corresponding to the masked graphic feature.
[0208] In a possible implementation, the target object is a target object (foreground object) in a sample image. Accordingly, the step of the server performing feature mask processing on the first graphic feature among the multiple graphic features is as follows: the server performs feature mask processing on the image feature in the first graphic feature, that is, the server sets the feature vector of the image feature to 0, thereby performing feature mask processing on the image feature. For example, continue to refer to Figure 5 The server trains the image processing model by using masked region classification, that is, the server masks the foreground object of the sample image and predicts the foreground object through the initial image processing model to train the image processing model. For example, the sample image is an image of a man holding a dog. The server masks the dog in the image and predicts the dog through the initial image processing model.
[0209] In the disclosed embodiment, the server masks the image features of the target object and predicts the position of the target object to train the image processing model, so that the image processing model can fully utilize the feature information of the sample graphic data and improve the utilization rate of the visual information in the image, thereby improving the reasoning ability of the image processing model.
[0210] In another possible implementation, the target object is target text information (such as vocabulary) in the sample text. Accordingly, the step of the server performing feature masking on the first graphic feature among the multiple graphic features is as follows: the server performs feature masking on the text feature in the first graphic feature, that is, the server sets the feature vector of the text feature to 0, thereby performing feature masking on the text feature. For example, continue to refer to Figure 5The server trains the image processing model by using masked language modeling, that is, the server masks the target text information of the sample text and predicts the target text information through the initial image processing model to train the image processing model. For example, if the sample text is "a man is taking a photo with a dog in his arms", the server masks the word "dog" in the text and predicts the word "dog" through the initial image processing model.
[0211] In the disclosed embodiment, the server masks the text features of the target object and predicts the position of the target object to train the image processing model, so that the image processing model can fully utilize the feature information of the sample image and text data, improve the utilization rate of the visual information in the image, and thus improve the reasoning ability of the image processing model.
[0212] It should be noted that the server randomly selects the target object with a preset probability through the initial image processing model, and then performs feature mask processing on the target object. For example, the server randomly selects the target text information (such as vocabulary) in the sample text with a probability of 15% and performs feature mask processing on it, or the server randomly selects the target object (foreground object) in the sample image with a probability of 15% and performs feature mask processing on it.
[0213] In the second case, the server masks the position features of the target object through the initial image processing model. Accordingly, the server performs feature masking processing based on the graphic features and the position features as follows: the server performs feature masking processing on the first position feature among multiple position features.
[0214] Since there are multiple target objects in the sample graphic data, the position feature includes multiple position features corresponding to the multiple target objects, wherein the first position feature is the position feature corresponding to the target object selected by the server. The server can randomly select one or more target objects, mask the first position feature of the target object, and thus predict the masked first position feature, that is, predict the position of the target object in the sample graphic data.
[0215] For example, the server randomly selects a target object in the sample image with a probability of 50%, and performs masking processing on the first position feature corresponding to the target object.
[0216] Step 605: The server predicts the second position information of the target object through the initial image processing model based on the remaining features.
[0217] The second position information is the position of the target object in the sample image data, which is the prediction result of the initial image processing model. Depending on the masked features, the server predicts the second position information of the target through the initial image processing model, which can be divided into the following cases, including:
[0218] In the first case, the obscured feature is the first graphic feature, and the server can obscured the graphic feature of the target object through the initial image processing model, and predict the second position information corresponding to the graphic feature by predicting the obscured graphic feature. Accordingly, the step of the server predicting the second position information of the target object through the initial image processing model is: the server predicts the position of the target object corresponding to the obscured first graphic feature through the initial image processing model based on the position feature and the remaining graphic features.
[0219] Depending on the target object, the server predicts the position of the target object corresponding to the obscured first graphic feature through the initial image processing model, which can be divided into multiple situations. For example, the target object is a target object in a sample image, and the server predicts the image features corresponding to the target object through the initial image processing model, and then predicts the position of the target object in the sample image. When the server trains the initial image processing model in this way, the regression loss function and the cross entropy loss function can be used to train the model, which are not specifically limited here. One thing that needs to be explained is that the server can also predict the category of the target object, that is, if the target object is an orange cat, the server can predict that the target object is a cat through the initial image processing model.
[0220] For another example, the target object is the target text information in the sample text. The server predicts the text features corresponding to the target object through the initial image processing model, and then predicts the position of the target object in the sample text. When the server trains the initial image processing model in this way, the cross entropy loss function can be used to train the model, which is not specifically limited here.
[0221] In the disclosed embodiment, the server masks the graphic features of the target object and predicts the position of the target object through the remaining graphic features and position features of the sample graphic data, thereby training the image processing model so that the image processing model can fully utilize the feature information of the sample graphic data, improve the utilization rate of the visual information in the image, and thus improve the reasoning ability of the image processing model.
[0222] In the second case, the masked feature is the first position feature, and the server can mask the position feature of the target object through the initial image processing model, thereby predicting the second position information corresponding to the position feature. Figure 5 The server trains the image processing model by target position regression. Correspondingly, the server predicts the second position information of the target object through the initial image processing model as follows: the server predicts the second position information of the target object corresponding to the obscured first position feature through the initial image processing model based on the image and text features and the remaining position features.
[0223] For example, the network structure of the initial image processing model includes MLP (Multilayer perceptron, fully connected neural network), and the server predicts the second position information of the target object corresponding to the masked first position feature through MLP. When the server trains the initial image processing model in this way, the regression loss function can be used to train the model, which is not specifically limited here.
[0224] In the disclosed embodiment, the server masks the position features of the target object, predicts the position of the target object through the graphic features and the remaining position features, and trains the image processing model from multiple dimensions, so that the image processing model can fully utilize the feature information of the sample graphic data, improve the utilization rate of the visual information in the image, and thus improve the reasoning ability of the image processing model.
[0225] Step 606: The server updates the initial image processing model based on the first position information and the second position information to obtain an image processing model.
[0226] Step 606 is the same as step 406 and will not be described again.
[0227] In the disclosed embodiment, the server masks the features of the target object (graphic features or position features), predicts the position of the target object through the features of the remaining graphic data, and trains the image processing model from multiple dimensions, so that the image processing model can fully utilize the feature information of the sample graphic data, improve the utilization rate of the visual information in the image, and thus improve the reasoning ability of the image processing model.
[0228] Figure 7 : is a flow chart of a training method for an image processing model according to an exemplary embodiment. The embodiment of the present disclosure is executed by a server. The method includes the following steps:
[0229] Step 701: The server obtains sample graphic and text data.
[0230] Step 702: The server determines the graphic features of the sample graphic data.
[0231] Step 703: The server determines the location characteristics of the sample graphic data.
[0232] Steps 701-703 are the same as steps 401-403 and will not be repeated here.
[0233] Step 704: The server obtains the bounding boxes corresponding to any two second graphic features.
[0234] In the sample image and text data, there are multiple target objects, and the image and text features include multiple image and text features corresponding to the multiple target objects. The server can predict the second position information through the initial image processing model, and the second position information is the position relationship between any two target objects in the sample image and text data. For example, continue to refer to Figure 5 The server trains the image processing model by adopting a spatial relationship classification method, that is, by obtaining the bounding boxes corresponding to any two second image and text features, the positional relationship between the corresponding two target objects is determined, thereby training the image processing model.
[0235] The bounding box is the bounding box of the target object corresponding to the second graphic feature, any two second graphic features correspond to two target objects, and the two target objects are both target objects in the sample image, or are both target text information in the sample text. The server randomly selects two target objects, obtains the position information of each of the two target objects, and obtains the bounding boxes corresponding to the target objects based on the respective position information.
[0236] Step 705: The server determines the intersection-over-union ratio of the two bounding boxes.
[0237] The intersection-over-union ratio is the ratio of the intersection to the union of two bounding boxes, the intersection of two bounding boxes is the overlapping area of the two bounding boxes, and the union of two bounding boxes is the total area of the two bounding boxes.
[0238] Step 706: Based on the IOR and the initial image processing model, the server predicts the positional relationship (second position information) of the target objects corresponding to the two bounding boxes.
[0239] From the above definition of IoU, it can be seen that the value of IoU is distributed between 0 and 1, and most of the values of IoU are low. It is difficult to directly use the regression loss function to train the initial image processing model. Therefore, the server can classify the values of IoU, and then classify and predict the positional relationship between any two target objects.
[0240] For example, the server divides the values of the intersection-over-union ratio into five categories, which are 0-0.2, 0.2-0.4, 0.4-0.6, 0.6-0.8, and 0.8-1, respectively, to indicate the degree of overlap of two bounding boxes, that is, the positional relationship between the two target objects corresponding to the two bounding boxes. When the server trains the initial image processing model in this way, the cross entropy loss function can be used to train the model, which is not specifically limited here.
[0241] Step 707: The server updates the initial image processing model based on the first position information and the second position information to obtain an image processing model.
[0242] Step 707 is the same as step 405 and will not be described in detail here.
[0243] In the disclosed embodiment, the server predicts the positional relationship between any two target objects through the graphic features and position features of the sample graphic data, and trains the image processing model from multiple dimensions, so that the image processing model can make full use of the feature information of the sample graphic data, improve the utilization rate of the visual information in the image, and thus improve the reasoning ability of the image processing model.
[0244] Figure 8 This is a flow chart of an image processing method according to an exemplary embodiment. The disclosed embodiment is executed by a server, and the method includes the following steps:
[0245] Step 801: The server obtains target graphic data.
[0246] The target image and text data includes the target image and target text, and the target text includes question information. For example, the target image of the target sample image is "a man wearing glasses and a woman without glasses", and the target text is the question information "who is wearing glasses in the picture". After the image processing model is trained, the server applies the model to downstream tasks, and the target sample image and text are image and text data related to the downstream tasks.
[0247] The method of obtaining the target sample image and text in step 801 is the same as the method of obtaining the sample image and text data in step 401, and will not be repeated here.
[0248] Step 802: The server determines target image features of the target image, target text features of the target text, and target position features based on the image processing model.
[0249] The method for the server to determine the target image features and the target text features of the target image and the target text in this step is the same as the method for determining the graphic features of the sample graphic data in step 402, and will not be repeated here.
[0250] Among them, the target position feature is used to indicate the position of the target object in the target image and the target text, and the target object includes at least one of the target object in the target image and the target text information in the target text. Accordingly, the process of determining the target position feature is: the server determines at least one of the third position feature of the target image and the fourth position feature of the target text, the third position feature is used to indicate the position of the target object in the target image, and the fourth position feature is used to indicate the position of the target text information in the target text; at least one of the third position feature and the fourth position feature is determined as the target position feature.
[0251] The method for the server to determine the target location feature is the same as the method for determining the location feature of the sample graphic data in step 403, and will not be described in detail here.
[0252] Step 803: The server determines the image processing result based on the target image features, target text features and target position features.
[0253] The image processing result is the answer information corresponding to the question information determined based on the target image, and the target image processing model is trained based on the image and text features and position features of the sample image and text data.
[0254] When executing downstream tasks, the server directly calls the trained image processing model, which processes the target image and target text to obtain image processing results. For example, if the downstream task is visual question answering, the server calls the image processing model and outputs the answer information corresponding to the target text based on the input target image and target text (question information).
[0255] In one possible implementation, the server calls a trained image processing model to perform feature masking on the features of the input target image data, thereby predicting the masked features, so as to more fully understand the information in the target image data, and output the answer information corresponding to the target text based on the information in the target image data. Depending on the features, the server performs feature masking to predict the masked features and then outputs the answer information corresponding to the target text, which can be divided into the following cases, including:
[0256] In the first case, when the masked feature is a target image feature or a target text feature, the server performs feature masking processing accordingly to predict the masked feature, and then outputs the answer information corresponding to the target text. The process is as follows: the server performs masking processing on some features of the target image feature and the target text feature through the image processing model; based on the remaining features, the image processing model is used to predict the position information corresponding to the masked feature; based on the position information corresponding to the masked feature, the image processing result is determined.
[0257] Among them, according to different features, the process of feature masking processing performed by the server can be divided into two types, including: one is that the server performs feature masking processing on target image features, and the other is that the server performs feature masking processing on target text features.
[0258] In the second case, when the masked feature is a target position feature, the server performs feature masking processing accordingly to predict the masked feature, and then outputs the answer information corresponding to the target text. The process is as follows: feature masking processing is performed on the third position feature among multiple target position features; the server predicts the position information corresponding to the masked third position feature based on the remaining features through the image processing model; and the image processing result is determined based on the position information corresponding to the masked third position feature.
[0259] In the above two cases, the server performs feature masking processing to predict the masked feature method, which is the same as the method in steps 604-605 where the server performs feature masking processing based on graphic features and position features and predicts the second position information of the target object based on the remaining features, and will not be repeated here.
[0260] In a possible implementation, there are multiple target objects in the target graphic data, and the server can predict the positional relationship between any two target objects in the target graphic data through an image processing model, thereby more fully understanding the positional information in the target graphic data, and outputting the answer information corresponding to the target text based on the positional information. Accordingly, the server predicts the positional relationship between any two target objects in the target graphic data, and then outputs the answer information corresponding to the target text in the following process: the server obtains the bounding boxes corresponding to any two target objects; determines the intersection-and-union ratio of the two bounding boxes, which is the ratio of the intersection to the union of the two bounding boxes; based on the intersection-and-union ratio, predicts the positional relationship between the two target objects through an image processing model; and determines the image processing result based on the positional relationship between the two target objects.
[0261] The method for the server to predict the positional relationship between any two target objects in the target graphic data is the same as the method for the server to predict the positional relationship between any two target objects in the sample graphic data in steps 704-706, and will not be repeated here.
[0262] In the disclosed embodiment, the server calls a trained image processing model to process the target image and text data. Since the image processing model is trained through the image and text features and position features of the sample image and text data, it has strong reasoning ability and can make full use of the feature information of the target image and text data. Therefore, the image processing result obtained by the image processing model is more accurate, and this method improves the accuracy of image processing.
[0263] Fig. 9 This is a flow chart of an image processing method according to an exemplary embodiment. The disclosed embodiment is executed by a server, and the method includes the following steps:
[0264] Step 901: The terminal obtains target graphic data.
[0265] The method of obtaining the target sample image and text in step 901 is the same as the method of obtaining the sample image and text data in step 801, and will not be repeated here.
[0266] Step 902: The terminal sends the target graphic data to the server.
[0267] Step 903: The server receives the target graphic data.
[0268] Step 904: The server processes the target image and target text based on the image processing model to obtain an image processing result.
[0269] The way of processing the target image and target text in this step is the same as the way of processing the sample image and sample text in the process of training the model, and will not be repeated here.
[0270] Step 905: The server sends the image processing result to the terminal.
[0271] Step 906: The terminal receives the image processing result.
[0272] In the disclosed embodiment, the server calls a trained image processing model to process the target image and text data. Since the image processing model is trained through the image and text features and position features of the sample image and text data, it has strong reasoning ability and can make full use of the feature information of the target image and text data. Therefore, the image processing result obtained by the image processing model is more accurate, and this method improves the accuracy of image processing.
[0273] Fig.10 is a structural diagram of a training device for an image processing model according to an exemplary embodiment. Fig.10 , the device comprises:
[0274] The first acquisition module 1001 is used to acquire sample image and text data, the sample image and text data includes a sample image and a sample text, and first position information of a target object marked in the sample image and text data;
[0275] Determination module 1002, used to determine the graphic features and position features of the sample graphic data, the graphic features include the image features of the sample image and the text features of the sample text, and the position features are used to indicate the position of the target object in the sample graphic data;
[0276] Prediction module 1003, used to predict second position information of the target object through an initial image processing model based on the image and text features and the position features;
[0277] The updating module 1004 is used to update the initial image processing model based on the first position information and the second position information to obtain the image processing model.
[0278] In a possible implementation, the determining module 1002 includes:
[0279] A first determining unit, used to determine at least one of a first position feature of the sample image and a second position feature of the sample text, wherein the first position feature is used to indicate the position of the target object in the sample image, and the second position feature is used to indicate the position of the target text information in the sample text;
[0280] The second determining unit is used to determine at least one of the first position feature and the second position feature as the position feature of the sample graphic data.
[0281] In another possible implementation, the prediction module 1003 includes:
[0282] A first prediction unit is used for, when the position feature includes the first position feature, superimposing the image feature and the first position feature, and predicting second position information of the target object based on the superimposed image feature and text feature through an initial image processing model;
[0283] A second prediction unit is used for, when the position feature includes a second position feature, superimposing the text feature and the second position feature, and predicting second position information of the target object based on the text feature and the image feature obtained by superimposing;
[0284] The third prediction unit is used to predict the second position information of the target object based on the superimposed text feature and the superimposed image feature when the position feature includes the first position feature and the second position feature, and the superimposed image feature and the first position feature, and the superimposed text feature and the second position feature.
[0285] In another possible implementation, the prediction module 1003 includes:
[0286] A first processing unit, configured to perform feature masking processing on a first image and text feature among the plurality of image and text features;
[0287] The fourth prediction unit is used to predict the position of the target object corresponding to the masked first image and text features through the initial image processing model based on the remaining image and text features and the position features.
[0288] In another possible implementation, the prediction module 1003 includes:
[0289] An acquisition unit, used to acquire a bounding box corresponding to any two second graphic features, where the bounding box is a bounding box of a target object corresponding to the second graphic features;
[0290] A third determining unit is used to determine an intersection-and-union ratio of two bounding boxes, where the intersection-and-union ratio is a ratio of the intersection to the union of the two bounding boxes;
[0291] The fifth prediction unit is used to predict the positional relationship of the target objects corresponding to the two bounding boxes through the initial image processing model based on the intersection-over-union ratio.
[0292] In another possible implementation manner, the first processing unit is configured to perform feature masking processing on image features in the first graphic feature; or perform feature masking processing on text features in the first graphic feature.
[0293] In another possible implementation, the prediction module 1003 includes:
[0294] A second processing unit is used to perform feature mask processing on a first position feature among the multiple position features;
[0295] The sixth prediction unit is used to predict the second position information of the target object corresponding to the masked first position feature through the initial image processing model based on the graphic feature and the remaining position features.
[0296] In another possible implementation, the prediction module 1003 includes:
[0297] a fourth determining unit, configured to determine a first sample feature based on the image-text feature, the position feature, and at least one marker feature, wherein the marker feature is between the image feature and the text feature, and the at least one marker feature is used to identify the positions of the image feature and the text feature;
[0298] The seventh prediction unit is used to predict the second position information of the target object through the initial image processing model based on the first sample feature.
[0299] In another possible implementation, the fourth determination unit is used to fuse the image feature, the text feature and the position feature to determine the second sample feature; the first marker feature, the second marker feature and the third marker feature are respectively inserted into the second sample feature to obtain the first sample feature; the first marker feature is before the image feature, and the first marker feature is a global feature of the sample image; the second marker feature is between the text features, and the second marker feature is a global feature of the sample text; the third marker feature is located at the end of the second sample feature, and the third marker feature is a terminator.
[0300] In another possible implementation, the device further includes:
[0301] A second acquisition module is used to acquire test sample data;
[0302] The adjustment module is used to adjust the model parameters of the image processing model based on the test sample data.
[0303] In the disclosed embodiments, the position information of the target object is predicted by the graphic features and position features of the sample graphic data, thereby training the image processing model so that the image processing model can make full use of the feature information of the sample graphic data, improve the utilization rate of the visual information in the image, and thus improve the reasoning ability of the image processing model.
[0304] Fig.11 is a schematic diagram of the structure of an image processing device according to an exemplary embodiment. Fig.11 , the device comprises:
[0305] An acquisition module 1101 is used to acquire a target sample image and text, where the target sample image and text include a target image and a target text, and the target text includes question information;
[0306] Processing module 1102 is used to process the target image and target text based on the image processing model to obtain an image processing result. The image processing result is answer information corresponding to the question information determined based on the target image. The target image processing model is trained based on the image and text features and position features of the sample image and text data.
[0307] In a possible implementation, the processing module 1102 includes:
[0308] A first determining unit is used to determine a target image feature of a target image, a target text feature of a target text, and a target position feature based on an image processing model, wherein the target position feature is used to indicate a position of a target object in the target image and the target text;
[0309] The second determining unit is used to determine the image processing result based on the target image feature, the target text feature and the target position feature.
[0310] In another possible implementation, the first determination unit is used to determine at least one of a third position feature of the target image and a fourth position feature of the target text, where the third position feature is used to indicate the position of the target object in the target image and the fourth position feature is used to indicate the position of the target text information in the target text; and at least one of the third position feature and the fourth position feature is determined as the target position feature.
[0311] In another possible implementation, the second determination unit is used to mask some features of the target image features and the target text features, and based on the remaining features, predict the position information corresponding to the masked features through the image processing model; and determine the image processing results based on the position information corresponding to the masked features.
[0312] In another possible implementation manner, the second determining unit is configured to perform feature masking processing on the target image features; or perform feature masking processing on the target text features.
[0313] In another possible implementation, the second determination unit is used to perform feature masking processing on a third position feature among multiple target position features; based on the remaining features, predict the position information corresponding to the masked third position feature through an image processing model; and determine the image processing result based on the position information corresponding to the masked third position feature.
[0314] In another possible implementation, the second determination unit is used to obtain the bounding boxes corresponding to any two target objects; determine the intersection-and-union ratio of the two bounding boxes, where the intersection-and-union ratio is the ratio of the intersection and the union of the two bounding boxes; based on the intersection-and-union ratio, predict the positional relationship between the two target objects through an image processing model; and determine the image processing result based on the positional relationship between the two target objects.
[0315] In the disclosed embodiment, the target image and target text are processed by an image processing model. Since the image processing model is trained by the image and text features and position features of sample image and text data, the image processing result obtained by the image processing model is more accurate. This method improves the accuracy of image processing.
[0316] Fig.12 The structure block diagram of a terminal 1200 provided by an exemplary embodiment of the present disclosure is shown. The terminal 1200 may be: a smart phone, a tablet computer, an MP3 player (Moving Picture Experts Group Audio Layer III, Moving Picture Experts Group Audio Layer 3), an MP4 (Moving Picture Experts Group Audio Layer IV, Moving Picture Experts Group Audio Layer 4) player, a laptop computer or a desktop computer. The terminal 1200 may also be called a user device, a portable terminal, a laptop terminal, a desktop terminal or other names.
[0317] Typically, the terminal 1200 includes: a processor 1201 and a memory 1202 .
[0318] The processor 1201 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 1201 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 1201 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 1201 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 1201 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.
[0319] The memory 1202 may include one or more computer-readable storage media, which may be non-transitory. The memory 1202 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1202 is used to store at least one instruction, which is used to be executed by the processor 1201 to implement the training method and image processing method of the image processing model provided in the method embodiment of the present disclosure.
[0320] In some embodiments, the terminal 1200 may further optionally include: a peripheral device interface 1203 and at least one peripheral device. The processor 1201, the memory 1202 and the peripheral device interface 1203 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 1203 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 1204, a touch display screen 1205, a camera 1206 camera assembly 1206, an audio circuit 1207, a positioning assembly 1208 and a power supply 1209.
[0321] The peripheral device interface 1203 may be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 1201 and the memory 1202. In some embodiments, the processor 1201, the memory 1202, and the peripheral device interface 1203 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 1201, the memory 1202, and the peripheral device interface 1203 may be implemented on a separate chip or circuit board, which is not limited in the embodiments of the present disclosure.
[0322] The radio frequency circuit 1204 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 1204 communicates with the communication network and other communication devices through electromagnetic signals. The radio frequency circuit 1204 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the radio frequency circuit 1204 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The radio frequency circuit 1204 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes but is not limited to: the World Wide Web, a metropolitan area network, an intranet, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 1204 may also include circuits related to NFC (Near Field Communication), which is not limited in the embodiments of the present disclosure.
[0323] The display screen 1205 is used to display a UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 1205 is a touch display screen, the display screen 1205 also has the ability to collect touch signals on the surface or above the surface of the display screen 1205. The touch signal can be input to the processor 1201 as a control signal for processing. At this time, the display screen 1205 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 1205 can be one, and the front panel of the terminal 1200 is set; in other embodiments, the display screen 1205 can be at least two, which are respectively set on different surfaces of the terminal 1200 or are folded; in some other embodiments, the display screen 1205 can be a flexible display screen, which is set on the curved surface or folded surface of the terminal 1200. Even, the display screen 1205 can also be set to a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 1205 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).
[0324] The camera assembly 1206 is used to capture images or videos. Optionally, the camera assembly 1206 includes a front camera and a rear camera. Typically, the front camera is arranged on the front panel of the terminal, and the rear camera is arranged on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize the panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 1206 may also include a flash. The flash can be a monochrome temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.
[0325] The audio circuit 1207 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals and input them into the processor 1201 for processing, or input them into the radio frequency circuit 1204 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there may be multiple microphones, which are respectively arranged at different parts of the terminal 1200. The microphone may also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 1201 or the radio frequency circuit 1204 into sound waves. The speaker may be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 1207 may also include a headphone jack.
[0326] The positioning component 1208 is used to locate the current geographical location of the terminal 1200 to implement navigation or LBS (Location Based Service). The positioning component 1208 can be a positioning component based on the US GPS (Global Positioning System), China's Beidou system, Russia's Grenas system, or Russia's European Union's Galileo system.
[0327] The power supply 1209 is used to power various components in the terminal 1200. The power supply 1209 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 1209 includes a rechargeable battery, the rechargeable battery can be a battery that supports wired charging or a wireless charging battery. A wired charging battery is a battery that is charged through a wired line, and a wireless charging battery is a battery that is charged through a wireless coil. The rechargeable battery can also be used to support fast charging technology.
[0328] In some embodiments, the terminal 1200 further includes one or more sensors 1120 , including but not limited to: an acceleration sensor 1211 , a gyroscope sensor 1212 , a pressure sensor 1213 , a fingerprint sensor 1214 , an optical sensor 1215 , and a proximity sensor 1216 .
[0329] The acceleration sensor 1211 can detect the magnitude of acceleration on the three coordinate axes of the coordinate system established by the terminal 1200. For example, the acceleration sensor 1211 can be used to detect the components of gravity acceleration on the three coordinate axes. The processor 1201 can control the touch display screen 1205 to display the user interface in a horizontal view or a vertical view according to the gravity acceleration signal collected by the acceleration sensor 1211. The acceleration sensor 1211 can also be used for collecting game or user motion data.
[0330] The gyro sensor 1212 can detect the body direction and rotation angle of the terminal 1200, and the gyro sensor 1212 can cooperate with the acceleration sensor 1211 to collect the user's 3D actions on the terminal 1200. The processor 1201 can implement the following functions based on the data collected by the gyro sensor 1212: motion sensing (such as changing the UI according to the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.
[0331] The pressure sensor 1213 can be set on the side frame of the terminal 1200 and / or the lower layer of the touch display screen 1205. When the pressure sensor 1213 is set on the side frame of the terminal 1200, it can detect the user's holding signal of the terminal 1200, and the processor 1201 performs left and right hand recognition or shortcut operation according to the holding signal collected by the pressure sensor 1213. When the pressure sensor 1213 is set on the lower layer of the touch display screen 1205, the processor 1201 controls the operability controls on the UI interface according to the user's pressure operation on the touch display screen 1205. The operability controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.
[0332] The fingerprint sensor 1214 is used to collect the user's fingerprint, and the processor 1201 identifies the user's identity based on the fingerprint collected by the fingerprint sensor 1214, or the fingerprint sensor 1214 identifies the user's identity based on the collected fingerprint. When the user's identity is identified as a trusted identity, the processor 1201 authorizes the user to perform relevant sensitive operations, which include unlocking the screen, viewing encrypted information, downloading software, paying, and changing settings. The fingerprint sensor 1214 can be set on the front, back, or side of the terminal 1200. When a physical button or a manufacturer logo is set on the terminal 1200, the fingerprint sensor 1214 can be integrated with the physical button or the manufacturer logo.
[0333] The optical sensor 1215 is used to collect the ambient light intensity. In one embodiment, the processor 1201 can control the display brightness of the touch display screen 1205 according to the ambient light intensity collected by the optical sensor 1215. Specifically, when the ambient light intensity is high, the display brightness of the touch display screen 1205 is increased; when the ambient light intensity is low, the display brightness of the touch display screen 1205 is reduced. In another embodiment, the processor 1201 can also dynamically adjust the shooting parameters of the camera assembly 1206 according to the ambient light intensity collected by the optical sensor 1215.
[0334] The proximity sensor 1216, also called a distance sensor, is usually arranged on the front panel of the terminal 1200. The proximity sensor 1216 is used to collect the distance between the user and the front of the terminal 1200. In one embodiment, when the proximity sensor 1216 detects that the distance between the user and the front of the terminal 1200 is gradually decreasing, the processor 1201 controls the touch display screen 1205 to switch from the screen-on state to the screen-off state; when the proximity sensor 1216 detects that the distance between the user and the front of the terminal 1200 is gradually increasing, the processor 1201 controls the touch display screen 1205 to switch from the screen-off state to the screen-on state.
[0335] Those skilled in the art will understand that Fig.12 The structure shown in the figure does not constitute a limitation on the terminal 1200, and the terminal 1200 may include more or fewer components than those shown in the figure, or combine certain components, or adopt a different component arrangement.
[0336] In an exemplary embodiment, the electronic device is provided as a server. Fig.13 It is a structural diagram of a server provided by an embodiment of the present disclosure. The server 1300 may have relatively large differences due to different configurations or performances, and may include one or more processors (Central Processing Units, CPU) 1301 and one or more memories 1302, wherein at least one computer program is stored in the memory 1302, and at least one computer program is loaded and executed by the processor 1301 to implement the training method and image processing method of the image processing model provided by the above-mentioned various method embodiments. Of course, the server may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output. The server may also include other components for implementing device functions, which will not be repeated here.
[0337] In an exemplary embodiment, a computer-readable storage medium is also provided, in which at least one instruction is stored, and the at least one instruction is loaded and executed by a terminal to implement the training method of the image processing model and the image processing method in the above-mentioned embodiment. The computer-readable storage medium can be a memory. For example, the computer-readable storage medium can be a ROM (Read-Only Memory), a RAM (Random Access Memory), a CD-ROM (Compact Disc Read-Only Memory), a magnetic tape, a floppy disk, and an optical data storage device, etc.
[0338] In an exemplary embodiment, a computer program product is also provided, including a computer program, and when the computer program code is executed by a processor, the computer implements the training method of the image processing model and the image processing method in the above-mentioned embodiment.
[0339] In an exemplary embodiment, the computer program involved in the embodiments of the present disclosure may be deployed and executed on one computer device, or on multiple computer devices located at one location, or on multiple computer devices distributed at multiple locations and interconnected through a communication network. Multiple computer devices distributed at multiple locations and interconnected through a communication network may constitute a blockchain system.
[0340] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.
[0341] The above description is only an optional embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure shall be included in the protection scope of the present disclosure.
Claims
1. A training method for an image processing model, characterized in that: The method comprises: Acquire sample graphic data, the sample graphic data including sample images and sample texts, and first position information of a target object is marked in the sample graphic data; the number of the target objects is multiple; Determine the graphic features and position features of the sample graphic data, wherein the graphic features include image features of the sample image and text features of the sample text, and the position features are used to indicate the position of the target object in the sample graphic data; The graphic features include multiple graphic features corresponding to multiple target objects; a first graphic feature among the multiple graphic features is subjected to feature masking processing; based on the remaining graphic features and the position features, a second position information of the target object corresponding to the masked first graphic feature is predicted through an initial image processing model; and / or, The position features include multiple position features corresponding to multiple target objects; a fifth position feature among the multiple position features is subjected to feature masking processing; based on the graphic features and the remaining position features, the second position information of the target object corresponding to the masked fifth position feature is predicted through the initial image processing model; Based on the first position information and the second position information, the initial image processing model is updated to obtain an image processing model.
2. The method according to claim 1, characterized in that: The target object includes at least one of a target object in the sample image and target text information in the sample text. The process of determining the position feature of the sample image and text data includes: Determine at least one of a first position feature of the sample image and a second position feature of the sample text, wherein the first position feature is used to indicate the position of the target object in the sample image, and the second position feature is used to indicate the position of the target text information in the sample text; At least one of the first position feature and the second position feature is determined as the position feature of the sample graphic data.
3. The method according to claim 2, characterized in that: The method further comprises: When the position feature includes the first position feature, superimposing the image feature and the first position feature, and predicting the second position information of the target object through the initial image processing model based on the superimposed image feature and the text feature; When the position feature includes the second position feature, superimposing the text feature and the second position feature, and predicting the second position information of the target text information through the initial image processing model based on the superimposed text feature and the image feature; When the position feature includes the first position feature and the second position feature, the image feature and the first position feature are superimposed, and the text feature and the second position feature are superimposed. Based on the superimposed text features and the superimposed image features, the second position information of the target object and the second position information of the target text information are predicted through the initial image processing model.
4. The method according to claim 1, characterized in that: The graphic features include multiple graphic features corresponding to the multiple target objects, and the second position information is the position relationship between any two target objects in the sample graphic data; The method further comprises: Obtaining bounding boxes corresponding to any two second graphic features, where the bounding boxes are bounding boxes of target objects corresponding to the second graphic features; Determine an intersection-of-union ratio of two bounding boxes, where the intersection-of-union ratio is a ratio of the intersection to the union of the two bounding boxes; Based on the intersection-over-union ratio, the positional relationship of the target objects corresponding to the two bounding boxes is predicted through the initial image processing model.
5. The method according to claim 1, characterized in that: The performing feature mask processing on the first graphic feature among the multiple graphic features includes at least one of the following implementation methods: Performing feature masking processing on the image features in the first graphic features; or, Perform feature masking processing on the text features in the first image-text features.
6. The method according to claim 1, characterized in that: The predicting the second position information of the target object by using an initial image processing model based on the graphic feature and the position feature includes: Determine a first sample feature based on the image-text feature, the position feature and at least one mark feature, wherein the at least one mark feature is used to identify a positional relationship between the image feature and the text feature; Based on the first sample feature, second position information of the target object is predicted by the initial image processing model.
7. The method according to claim 6, characterized in that: The determining of the first sample feature based on the graphic feature, the position feature and at least one mark feature includes: Fusing the image feature, the text feature and the position feature to determine a second sample feature; Inserting the first signature feature, the second signature feature and the third signature feature into the second sample feature respectively to obtain the first sample feature; The first signature feature is before the image feature, and the first signature feature is a global feature of the sample image; The second signature feature is before the text feature, and the second signature feature is a global feature of the sample text; The third marker feature is located at the end of the second sample feature, and the third marker feature is a terminator.
8. The method according to claim 1, characterized in that: The method further comprises: Get test sample data; Based on the test sample data, the model parameters of the image processing model are adjusted.
9. An image processing method, characterized in that: The method comprises: Acquire target graphic data, wherein the target graphic data includes a target image and a target text, and the target text includes question information; Based on the image processing model, determining a target image feature of the target image, a target text feature of the target text, and a target position feature, wherein the target position feature is used to represent the position of the target object in the target image and the target text; Masking some of the target image features and the target text features, and predicting position information corresponding to the masked features through the image processing model based on the remaining features; determining an image processing result based on the position information corresponding to the masked features; and / or, The number of the target objects is multiple, and the target position features include multiple target position features corresponding to the multiple target objects; a sixth position feature among the multiple target position features is subjected to feature masking processing; based on the remaining features, the position information corresponding to the masked sixth position feature is predicted through the image processing model; and an image processing result is determined based on the position information corresponding to the masked sixth position feature; The image processing result is answer information corresponding to the question information determined based on the target image, and the image processing model is trained based on the image and text features and position features of sample image and text data.
10. The method according to claim 9, characterized in that: The target object includes at least one of a target object in the target image and target text information in the target text, and the process of determining the target position feature includes: Determine at least one of a third position feature of the target image and a fourth position feature of the target text, wherein the third position feature is used to indicate the position of the target object in the target image, and the fourth position feature is used to indicate the position of the target text information in the target text; At least one of the third position feature and the fourth position feature is determined as the target position feature.
11. The method according to claim 9, characterized in that: The masking of some of the target image features and the target text features includes at least one of the following implementations: Performing feature mask processing on the target image features; or, Perform feature masking processing on the target text features.
12. The method according to claim 9, characterized in that: The number of the target objects is multiple; the method further includes: Get the bounding boxes corresponding to any two target objects; Determine an intersection-of-union ratio of two bounding boxes, where the intersection-of-union ratio is a ratio of the intersection to the union of the two bounding boxes; Based on the intersection-over-union ratio, predicting the positional relationship between the two target objects through the image processing model; An image processing result is determined based on the positional relationship between the two target objects.
13. A training device for an image processing model, characterized in that: The device comprises: A first acquisition module is used to acquire sample graphic data, the sample graphic data includes a sample image and a sample text, and first position information of a target object is marked in the sample graphic data; the number of the target objects is multiple; A determination module, used to determine the graphic features and position features of the sample graphic data, wherein the graphic features include the image features of the sample image and the text features of the sample text, and the position features are used to indicate the position of the target object in the sample graphic data; The graphic features include multiple graphic features corresponding to multiple target objects; the prediction module includes a first processing unit and a fourth prediction unit; The first processing unit is used to perform feature mask processing on a first image and text feature among the multiple image and text features; The fourth prediction unit is used to predict the second position information of the target object corresponding to the first image and text feature that is covered by the initial image processing model based on the remaining image and text features and the position features; and / or The position feature includes a plurality of position features corresponding to a plurality of target objects; the prediction module includes a second processing unit and a sixth prediction unit; The second processing unit is used to perform feature mask processing on a fifth position feature among the multiple position features; The sixth prediction unit is used to predict the second position information of the target object corresponding to the covered fifth position feature through the initial image processing model based on the graphic feature and the remaining position features; An updating module is used to update the initial image processing model based on the first position information and the second position information to obtain an image processing model.
14. The device according to claim 13, characterized in that: The target object includes at least one of a target object in the sample image and target text information in the sample text; The determining module comprises: A first determining unit, configured to determine at least one of a first position feature of the sample image and a second position feature of the sample text, wherein the first position feature is used to indicate the position of the target object in the sample image, and the second position feature is used to indicate the position of the target text information in the sample text; The second determining unit is used to determine at least one of the first position feature and the second position feature as the position feature of the sample graphic data.
15. The device according to claim 14, characterized in that: The prediction module comprises: A first prediction unit is configured to, when the position feature includes the first position feature, superimpose the image feature and the first position feature, and predict second position information of the target object through the initial image processing model based on the superimposed image feature and the text feature; A second prediction unit is used for, when the position feature includes the second position feature, superimposing the text feature and the second position feature, and predicting the second position information of the target text information through the initial image processing model based on the text feature and the image feature obtained by superimposing; A third prediction unit is used to, when the position feature includes the first position feature and the second position feature, superimpose the image feature and the first position feature, superimpose the text feature and the second position feature, and predict the second position information of the target object and the second position information of the target text information through the initial image processing model based on the superimposed text features and the superimposed image features.
16. The device according to claim 13, characterized in that: The graphic features include multiple graphic features corresponding to the multiple target objects, and the second position information is the position relationship between any two target objects in the sample graphic data; The prediction module comprises: An acquisition unit, configured to acquire a bounding box corresponding to any two second graphic features, wherein the bounding box is a bounding box of a target object corresponding to the second graphic features; a third determining unit, configured to determine an intersection-and-union ratio of two bounding boxes, wherein the intersection-and-union ratio is a ratio of the intersection to the union of the two bounding boxes; A fifth prediction unit is used to predict the positional relationship of the target objects corresponding to the two bounding boxes based on the intersection-over-union ratio and through the initial image processing model.
17. The device according to claim 13, characterized in that: The first processing unit is used to perform at least one of the following: performing feature masking processing on image features in the first graphic features; or performing feature masking processing on text features in the first graphic features.
18. The device according to claim 13, characterized in that: The prediction module comprises: a fourth determining unit, configured to determine a first sample feature based on the image-text feature, the position feature, and at least one mark feature, wherein the at least one mark feature is used to identify a positional relationship between the image feature and the text feature; A seventh prediction unit is used to predict second position information of the target object through the initial image processing model based on the first sample feature.
19. The device according to claim 18, characterized in that The fourth determination unit is used to fuse the image feature, the text feature and the position feature to determine the second sample feature; insert the first marker feature, the second marker feature and the third marker feature into the second sample feature respectively to obtain the first sample feature; the first marker feature is before the image feature, and the first marker feature is a global feature of the sample image; the second marker feature is before the text feature, and the second marker feature is a global feature of the sample text; the third marker feature is located at the end of the second sample feature, and the third marker feature is a terminator.
20. The device according to claim 13, characterized in that: The device also includes: A second acquisition module is used to acquire test sample data; An adjustment module is used to adjust the model parameters of the image processing model based on the test sample data.
21. An image processing device, characterized in that: The device comprises: An acquisition module, used for acquiring a target sample image and text, wherein the target sample image and text include a target image and a target text, and the target text includes question information; The processing module includes a first determining unit and a second determining unit; The first determining unit is used to determine a target image feature of the target image, a target text feature of the target text, and a target position feature based on an image processing model, wherein the target position feature is used to indicate a position of a target object in the target image and the target text; The second determination unit is used to perform masking processing on some features of the target image features and the target text features, and based on the remaining features, predict the position information corresponding to the masked features through the image processing model; determine the image processing result based on the position information corresponding to the masked features; and / or, the number of the target objects is multiple, and the target position features include multiple target position features corresponding to multiple target objects; perform feature masking processing on the sixth position feature of the multiple target position features; based on the remaining features, predict the position information corresponding to the masked sixth position feature through the image processing model; determine the image processing result based on the position information corresponding to the masked sixth position feature; The image processing result is answer information corresponding to the question information determined based on the target image, and the image processing model is trained based on the image and text features and position features of sample image and text data.
22. The device according to claim 21, characterized in that The target object includes at least one of a target object in the target image and target text information in the target text; The first determination unit is used to determine at least one of a third position feature of the target image and a fourth position feature of the target text, wherein the third position feature is used to indicate the position of the target object in the target image, and the fourth position feature is used to indicate the position of the target text information in the target text; and determine at least one of the third position feature and the fourth position feature as the target position feature.
23. The device according to claim 21, characterized in that The second determination unit is used to perform at least one of the following: performing feature masking processing on the target image feature; or performing feature masking processing on the target text feature.
24. The device according to claim 21, characterized in that The number of the target objects is multiple; the second determining unit is used to obtain the bounding boxes corresponding to any two target objects; determine the intersection-and-union ratio of the two bounding boxes, the intersection-and-union ratio being the ratio of the intersection to the union of the two bounding boxes; Based on the intersection-over-union ratio, predicting the positional relationship between the two target objects through the image processing model; An image processing result is determined based on the positional relationship between the two target objects.
25. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the training method of the image processing model as described in any one of claims 1 to 8 or the image processing method as described in any one of claims 9 to 12.
26. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by the processor of the terminal, the terminal is enabled to execute the training method of the image processing model as described in any one of claims 1 to 8 or the image processing method as described in any one of claims 9 to 12.
27. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, it implements the training method of the image processing model described in any one of claims 1 to 8 or the image processing method described in any one of claims 9 to 12.
