Data processing method and apparatus, model training method and apparatus, device, medium, and program product

Abstract

Embodiments of the present application disclose a data processing method and apparatus, a model training method and apparatus, a device, a medium, and a program product. The method comprises: acquiring a first image to be processed in a target task; dividing the first image, to obtain multiple sub-images, and acquiring a first position feature of each sub-image in the first image; for each sub-image, encoding the sub-image by means of an image encoding and decoding model, to obtain a first image feature of the sub-image, and fusing the first image feature and the first position feature, to obtain a second image feature of the sub-image; processing the second image features of the multiple sub-images by means of a trained task processing model, to obtain a third image feature; and decoding the third image feature by means of the image encoding and decoding model, to obtain a second image.

Description

Data processing method, model training method, device, equipment, medium and program product

[0001] The present application claims priority to the Chinese patent application No. 202410474106.9, filed on April 18, 2024, and entitled "Data processing method, model training method, device, equipment and medium". TECHNICAL FIELD

[0002] Embodiments of the present application relate to the technical field of computer, in particular to a data processing method, a model training method, a device, an equipment, a medium and a program product.

[0003] BACKGROUND

[0004] With the continuous development of artificial intelligence technology, the application of artificial intelligence model is also more and more extensive. Using artificial intelligence can realize the interaction task with the user, for example, the user can provide an image and propose the demand for the image, and the artificial intelligence model can process the image according to the demand of the user.

[0005] However, the size of the image in different fields is various, which leads to the difficulty of the artificial intelligence model to adapt to the data processing demand in different fields, so the adaptability of the data processing method is poor. SUMMARY

[0006] Embodiments of the present application provide a data processing method, a model training method, a device, an equipment, a medium and a program product, which can improve the adaptability of the data processing method. The technical solution is as follows:

[0007] In one aspect, the present application provides a data processing method, which is executed by an electronic device, and the method comprises:

[0008] obtaining a first image to be processed in a target task;

[0009] dividing the first image to obtain a plurality of sub-images, and obtaining a first position feature of each sub-image in the first image, wherein the resolution of each sub-image is equal to the target resolution of an image coding and decoding model that can be processed by a trained image.

[0010] For each sub-image, the image coding and decoding model is used to code the sub-image to obtain a first image feature of the sub-image, and the first image feature and the first position feature are fused to obtain a second image feature of the sub-image.

[0011] The second image features of the plurality of sub-images are processed by a trained task processing model to obtain a third image feature; and

[0012] The third image feature is decoded by the image coding and decoding model to obtain a second image.

[0013] In another aspect, an embodiment of the present application provides a model training method, executed by an electronic device, the method comprising:

[0014] obtaining a first sample image;

[0015] processing the first sample image based on a target task to obtain a target sample image and obtaining a target sample image feature of the target sample image;

[0016] dividing the first sample image to obtain a plurality of first sample sub-images and obtaining a first sample position feature of each first sample sub-image in the first sample image, wherein a resolution of each first sample sub-image is equal to a target resolution of an image that can be processed by a trained image coding and decoding model;

[0017] for each first sample sub-image, encoding the first sample sub-image by the image coding and decoding model to obtain a first sample image feature of the first sample sub-image, and fusing the first sample image feature and the first sample position feature to obtain a second sample image feature of the first sample sub-image;

[0018] processing the second sample image features of the plurality of first sample sub-images by an initial processing model to obtain third sample image features; and

[0019] training the initial processing model based on the third sample image features and the target sample image feature to obtain a task processing model for processing the target task.

[0020] In another aspect, an embodiment of the present application provides a data processing apparatus, the apparatus comprising:

[0021] an obtaining module configured to obtain a first image to be processed in a target task;

[0022] an image processing module configured to divide the first image to obtain a plurality of sub-images and obtain a first position feature of each sub-image in the first image, wherein a resolution of each sub-image is equal to a target resolution of an image that can be processed by a trained image coding and decoding model;

[0023] an encoding module configured to, for each sub-image, encode the sub-image by the image coding and decoding model to obtain a first image feature of the sub-image, and fuse the first image feature and the first position feature to obtain a second image feature of the sub-image;

[0024] The feature processing module is configured to process the second image features of the plurality of sub-images by using the trained task processing model to obtain third image features.

[0025] The decoding module is configured to decode the third image features by using the image coding model to obtain a second image.

[0026] In another aspect, the embodiments of the present application provide a model training device, and the device comprises:

[0027] The acquisition module is configured to acquire a first sample image, process the first sample image based on a target task to obtain a target sample image, and acquire a target sample image feature of the target sample image.

[0028] The image processing module is configured to divide the first sample image to obtain a plurality of first sample sub-images, and acquire a first sample position feature of each first sample sub-image in the first sample image, wherein the resolution of each first sample sub-image is equal to a target resolution of an image that can be processed by a trained image coding model.

[0029] The encoding module is configured to, for each first sample sub-image, encode the first sample sub-image by using the image coding model to obtain a first sample image feature of the first sample sub-image, and fuse the first sample image feature and the first sample position feature to obtain a second sample image feature of the first sample sub-image.

[0030] The feature processing module is configured to process the second sample image features of the plurality of first sample sub-images by using an initial processing model to obtain third sample image features.

[0031] The first training module is configured to train the initial processing model based on the third sample image features and the target sample image feature to obtain a task processing model for processing the target task.

[0032] In another aspect, the embodiments of the present application provide an electronic device, which comprises a processor and a memory, and the memory stores at least one computer program, the at least one computer program is loaded and executed by the processor to implement the operations performed by the data processing method or the model training method in the above aspects.

[0033] In another aspect, the embodiments of the present application provide a computer readable storage medium, and the computer readable storage medium stores at least one computer program, the at least one computer program is loaded and executed by a processor to implement the operations performed by the data processing method or the model training method in the above aspects.

[0034] In another aspect, an embodiment of the present application provides a computer program product comprising a computer program, which is loaded and executed by a processor to implement the operations performed by the data processing method or the model training method according to the above aspects.

[0035] BRIEF DESCRIPTION OF DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort on the basis of these drawings.

[0037] FIG. 1 is a schematic diagram of an implementation environment according to an embodiment of the present application;

[0038] FIG. 2 is a flowchart of a data processing method according to an embodiment of the present application;

[0039] FIG. 3A is a flowchart of another data processing method according to an embodiment of the present application;

[0040] FIG. 3B is a schematic diagram of the architecture of a data processing model according to an embodiment of the present application;

[0041] FIG. 4 is a flowchart of another data processing method according to an embodiment of the present application;

[0042] FIG. 5 is a schematic diagram of a multi-modal data sequence according to an embodiment of the present application;

[0043] FIG. 6 is a schematic diagram of a data processing method according to an embodiment of the present application;

[0044] FIG. 7 is a schematic diagram of an artificial intelligence-based interaction method according to an embodiment of the present application;

[0045] FIG. 8 is a flowchart of a model training method according to an embodiment of the present application;

[0046] FIG. 9 is a flowchart of a training method of a task processing model according to an embodiment of the present application;

[0047] FIG. 10 is a flowchart of another training method of a task processing model according to an embodiment of the present application;

[0048] FIG. 11 is a flowchart of a training method of an image coding and decoding model according to an embodiment of the present application;

[0049] FIG. 12 is a schematic diagram of a training method of an image coding and decoding model according to an embodiment of the present application;

[0050] FIG. 13 is a comparison diagram of reconstruction results of an image coding model according to an embodiment of the present application;

[0051] FIG. 14 is a flowchart of a pre-training method of an initial processing model according to an embodiment of the present application;

[0052] FIG. 15 is a comparison diagram of data processing results according to an embodiment of the present application;

[0053] FIG. 16 is a structural diagram of a data processing apparatus according to an embodiment of the present application;

[0054] FIG. 17 is a structural diagram of a model training apparatus according to an embodiment of the present application;

[0055] FIG. 18 is a structural diagram of another model training apparatus according to an embodiment of the present application;

[0056] FIG. 19 is a structural diagram of a terminal according to an embodiment of the present application;

[0057] FIG. 20 is a structural diagram of a server according to an embodiment of the present application.

[0058] Embodiment

[0059] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the drawings.

[0060] It can be understood that the terms "first", "second", and the like used in the present application can be used herein to describe various concepts, but unless specifically stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the present application, a first image can be referred to as a second image, and similarly, a second image can be referred to as a first image.

[0061] Wherein, at least one refers to one or more than one, for example, at least one image can be one image, two images, three images, or any integer greater than or equal to one image. A plurality of refers to two or more than two, for example, a plurality of images can be two images, three images, or any integer greater than or equal to two images. Each refers to each of at least one, for example, each image refers to each of a plurality of images, if the plurality of images is 3 images, each image refers to each of the 3 images.

[0062] It should be noted that the information (including but not limited to user equipment information, user personal information, etc.), data (including but not limited to data for analysis, stored data, displayed data, etc.) and signals (including but not limited to signals transmitted between user terminals and other devices, etc.) involved in the present application are fully authorized by users or relevant parties, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0063] The data processing method and the model training method provided by the embodiments of the present application can be executed by an electronic device. In some embodiments, the electronic device is a computer device, for example, a terminal or a server. In some embodiments, the server is a stand-alone physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud service, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, CDN (Content Delivery Network), and big data and artificial intelligence platform. In some embodiments, the terminal is a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, a smart voice interaction device, a smart home appliance, and a vehicle-mounted terminal, but is not limited thereto.

[0064] In some embodiments, the computer program related to the embodiments of the present application can be deployed on an electronic device for execution, or on multiple electronic devices located in one place, or on multiple electronic devices distributed in multiple places and interconnected through a communication network. The multiple electronic devices distributed in multiple places and interconnected through a communication network can constitute a blockchain system.

[0065] In some embodiments, FIG. 1 is a schematic diagram of an implementation environment provided by the embodiments of the present application. Referring to FIG. 1, the implementation environment includes a terminal 101 and a server 102. The terminal 101 and the server 102 are connected through a wireless or wired network.

[0066] The server 102 adopts the model training method provided by the embodiments of the present application to train an image coding and decoding model and a task processing model, and stores the trained image coding and decoding model and the task processing model. The image coding and decoding model is used for encoding and decoding images, and the task processing model is used for generating features related to target tasks.

[0067] In some embodiments, the terminal 101 sends a target task carrying the first image to the server 102, the server 102 receives the target task, encodes the first image through the image codec model to obtain image features, processes the image features through the task processing model to obtain processed image features, and decodes the processed image features through the image codec model to obtain a second image, which is the result of processing the first image according to the target task. The server 102 returns the second image to the terminal 101, and the terminal 101 displays the second image to the user.

[0068] In other embodiments, the server 102 sends the image codec model and the task processing model to the terminal 101, and the terminal 101 encodes the first image through the image codec model to obtain image features, processes the image features through the task processing model to obtain processed image features, and decodes the processed image features through the image codec model to obtain a second image for a target task carrying the first image.

[0069] In some embodiments, the terminal 101 is installed with a target application provided by the server 102, the server 102 distributes the image codec model and the task processing model to the target application, and the terminal 101 can process the image and data required to be processed in the target task through the image codec model and the task processing model in the target application to obtain a processing result.

[0070] FIG. 2 is a flowchart of a data processing method according to an embodiment of the present application. The embodiment of the present application is executed by an electronic device, which can be the terminal 101 or the server 102 in FIG. 1. Referring to FIG. 2, the method includes:

[0071] 201, obtaining a first image to be processed in a target task.

[0072] In some embodiments, the terminal 101 sends a target task carrying the first image to the server 102, the server 102 receives the target task, encodes the first image through the image codec model to obtain image features, processes the image features through the task processing model to obtain processed image features, and decodes the processed image features through the image codec model to obtain a second image, which is the result of processing the first image according to the target task. The server 102 returns the second image to the terminal 101, and the terminal 101 displays the second image to the user.

[0073] In some embodiments, the terminal 101 sends a target task carrying the first image to the server 102, the server 102 receives the target task, encodes the first image through the image codec model to obtain image features, processes the image features through the task processing model to obtain processed image features, and decodes the processed image features through the image codec model to obtain a second image, which is the result of processing the first image according to the target task. The server 102 returns the second image to the terminal 101, and the terminal 101 displays the second image to the user.

[0074] 202. Divide the first image into multiple sub-images, and obtain a first position feature of each sub-image in the first image, wherein the resolution of each sub-image is equal to the target resolution of the image that can be processed by the trained image encoding and decoding model.

[0075] The electronic device divides the first image based on the target resolution corresponding to the image coding and decoding model to obtain a plurality of sub-images, and the resolution of each sub-image is equal to the target resolution.

[0076] In an embodiment of the present application, the resolution of the first image can be any resolution, and the image coding and decoding model corresponds to a target resolution, which refers to the resolution of the image that can be processed by the image coding and decoding model, that is, the image coding and decoding model can only process images with a resolution equal to the target resolution.

[0077] If the resolution of the first image is greater than the target resolution, the image coding model cannot directly process the first image. In this case, the electronic device divides the first image based on the target resolution, that is, segments the first image, to obtain multiple sub-images, so that the resolution of each sub-image is equal to the target resolution. Then, the subsequent image coding model can encode and decode on a sub-image basis, so even if the resolution of the first image does not match the target resolution corresponding to the image coding model, the first image can still be processed.

[0078] Because sub-images are derived from the first image, their first image features contain the content of the sub-image within the first image, but do not contain information about the sub-image's location within the first image. Therefore, to preserve the location information of each sub-image within the first image, the electronic device also obtains a first position feature for each sub-image, which indicates the sub-image's location within the first image.

[0079] 203. For each sub-image, encode the sub-image using an image encoding and decoding model to obtain a first image feature of the sub-image, and fuse the first image feature with the first position feature to obtain a second image feature of the sub-image.

[0080] In an embodiment of the present application, the image encoding and decoding model is used to encode and decode an image. In some embodiments, the image encoding and decoding model includes an image encoding network and an image decoding network. The image encoding network is used to encode the image to obtain image features; the image decoding network is used to decode the image features to obtain the image.

[0081] The electronic device inputs the sub-image into an image encoding network in the image coding and decoding model, encodes the sub-image by the image encoding network, and outputs a first image feature of the sub-image after the sub-image is segmented.

[0082] Further, for each sub-image, the electronic device fuses the first image feature of the sub-image and the first position feature of the sub-image to obtain a second image feature of the sub-image. The second image feature contains not only the content of the sub-image in the first image but also the position information of the sub-image in the first image. Therefore, the second image features of the plurality of sub-images can fully represent the features of the entire first image, thereby avoiding the problem of loss of position information due to division of the image.

[0083] 204. The second image features of the plurality of sub-images are processed by the trained task processing model to obtain a third image feature.

[0084] For example, the target task includes deblurring of the image, and the task processing model can generate an image feature after deblurring. For another example, the target task includes style transfer of the image, and the task processing model can generate an image feature after style transfer.

[0085] After the electronic device obtains the second image feature of each sub-image, the second image features of the plurality of sub-images are input into the task processing model. The task processing model processes the second image features of the plurality of sub-images, for example, performs deep feature extraction, feature cross, feature mapping, etc. The task processing model outputs a third image feature, which is the feature of the second image required to be generated by the target task.

[0086] 205. The third image feature is decoded by the image coding and decoding model to obtain a second image.

[0087] After the electronic device obtains the third image feature, the third image feature is input into an image decoding network in the image coding and decoding model. The image decoding network outputs the second image obtained by decoding. The second image is the processing result required by the target task, that is, the image processed according to the target task.

[0088] The method provided in the embodiments of the present application can divide the image into a plurality of sub-images with a resolution equal to the target resolution in the case that the image required to be processed in the target task does not match the target resolution corresponding to the image coding and decoding model, so as to directly encode the sub-images by the image coding and decoding model. Moreover, the position feature of the sub-image is fused into the image feature of the sub-image, so as to reserve the position information of each sub-image in the complete image. Furthermore, the image feature of each sub-image is processed by the task processing model, and the processed image feature is decoded, so as to obtain the second image, that is, the processing result of the target task. Therefore, by dividing the image and reserving the position information, the image coding and decoding model can be applied to images with any resolution, and the adaptability of the data processing method is improved.

[0089] On the basis of the above-mentioned embodiments, when the resolution of the first image is greater than the target resolution, the electronic device can encode the first image after scaling the first image to the target resolution in addition to encoding the plurality of sub-images obtained by dividing the first image. For details, refer to the embodiment shown in FIG. 3A.

[0090] FIG. 3A is a flowchart of another data processing method provided in the embodiments of the present application. The embodiments of the present application are executed by an electronic device, which can be the terminal 101 or the server 102 in FIG. 1. Referring to FIG. 3A, the method comprises the following steps.

[0091] 301. Obtain a first image to be processed in a target task.

[0092] In some embodiments, the electronic device receives a processing request of the target task, the processing request of the target task carrying the first image, and the processing request of the target task being used to instruct to process the first image according to the target task. In the embodiments of the present application, the first image can be an image with any resolution.

[0093] 302. Divide the first image to obtain a plurality of sub-images, and obtain a first position feature of each sub-image in the first image, wherein the resolution of each sub-image is equal to a target resolution of an image that can be processed by a trained image coding and decoding model.

[0094] The image coding and decoding model can only process images with a resolution equal to the target resolution. In the case that the resolution of the first image is greater than the target resolution, the image coding and decoding model cannot directly process the first image. Therefore, before scaling the first image, the first image is divided into sub-images with a resolution equal to the target resolution in step 302, so that the method in the embodiments of the present application can process images with any size and aspect ratio.

[0095] In some embodiments, the electronic device determines a plurality of division strategies, the division strategies including a first division number, a second division number, and a division ratio; determines a target division strategy from the plurality of division strategies based on a length-width ratio of the first image, the division ratio in the target division strategy being closest to the length-width ratio; and divides the first image based on the target division strategy and the target resolution to obtain a plurality of sub-images.

[0096] The first division number refers to a division number of the image in the length direction, the second division number refers to a division number of the image in the width direction, and the division ratio refers to a ratio between the first division number and the second division number. For example, the plurality of division strategies include division strategy 1, division strategy 2, and division strategy 3, the division ratio in the division strategy 1 is 5:1, the division ratio in the division strategy 2 is 4:3, and the division ratio in the division strategy 3 is 16:9. The length-width ratio of the first image is 16:9, and therefore the division ratio in the division strategy 3 is closest to the length-width ratio of the first image, and the division strategy 3 is determined as the target division strategy.

[0097] After the target division strategy is determined, the first image can be divided based on the first division number, the second division number in the target division strategy, and the target resolution, so as to obtain a plurality of sub-images.

[0098] In the embodiments of the present application, a plurality of division strategies to be selected are provided, a division strategy that is most suitable for the length-width ratio of the first image is selected as the target division strategy based on the division ratio in the division strategy, and then the first image is divided based on the target division strategy, which is beneficial to improving the efficiency of dividing the image by providing the division strategies in advance.

[0099] In some embodiments, the electronic device divides the first image based on the target division strategy and the target resolution to obtain a plurality of sub-images, including: generating a target size based on the target division strategy and the target resolution; upsampling the first image based on the target size to obtain a first intermediate image; and dividing the first intermediate image according to the target division strategy to obtain a plurality of sub-images.

[0100] For example, the first division number in the target division strategy is Nh, the second division number is Nw, the target resolution is Ht x Wt, and the target size includes a length and a width, the length of the target size is equal to Nh*Ht, and the width of the target size is equal to Nw*Wt. The first image is upsampled to the first intermediate image, so that the length of the first intermediate image is equal to the length of the target size Nh*Ht, and the width of the first intermediate image is equal to the width of the target size Nw*Wt. Then, the first intermediate image is divided according to the target division strategy, that is, the first intermediate image is divided into Nh parts in the length direction and Nw parts in the width direction, so as to obtain a plurality of sub-images.

[0101] In the embodiments of the present application, a target size is generated based on a target division strategy and a target resolution, the target size being a size required for dividing the first image into sub-images of the target resolution according to the division strategy, and the first image is up-sampled into a first intermediate image of the target size, so as to ensure that the resolution of each sub-image is equal to the target resolution.

[0102] 303、For each sub-image, the image encoding and decoding model is used to encode the sub-image to obtain a first image feature of the sub-image, and the first image feature and the first position feature are fused to obtain a second image feature of the sub-image.

[0103] In some embodiments, the image encoding and decoding model includes an image encoding network and an image decoding network, and the image encoding network can be a VAE (Variational AutoEncoder), a ViT (Visual Tokenizer), or the like. The type of the image encoding network is not limited in the embodiments of the present application.

[0104] For any sub-image, the electronic device inputs the sub-image into the image encoding network in the image encoding and decoding model, encodes the sub-image by the image encoding network, and outputs a first image feature of the sub-image, the first image feature being an image feature of the sub-image.

[0105] Further, the first position feature of each sub-image is added to the first image feature of the sub-image to obtain a second image feature of the sub-image, so that the second image includes not only the content of the sub-image itself but also the position information of the sub-image in the complete first image.

[0106] In some embodiments, the electronic device obtains the first position feature of each sub-image in the first image, including: for any sub-image, determining coordinate information of the sub-image in the first image; and mapping the coordinate information based on a mapping parameter to obtain the first position feature of the sub-image, the mapping parameter being obtained by training. The mapping processing specifically refers to the operation of multiplying the mapping parameter and the coordinate information.

[0107] In some embodiments, the mapping parameter includes a left mapping parameter, a right mapping parameter, an upper mapping parameter, and a lower mapping parameter. The coordinate information includes a normalized horizontal coordinate and a normalized vertical coordinate, and the electronic device can determine the first position feature by using the following formula:

[0108] p = x c · l + (1 - x c) · r + y c · t + (1 - y c) · b

[0109] wherein p represents the first position feature, x c represents the horizontal coordinate, y c represents the vertical coordinate, l represents the left mapping parameter, r represents the right mapping parameter, t represents the top mapping parameter, and b represents the bottom mapping parameter. In some embodiments, the above mapping parameters are all vectors.

[0110] In some embodiments, the sub-image includes a plurality of image blocks, which are the smallest units in the image, such as pixels, etc. The first position feature of the sub-image includes the position feature of each image block in the sub-image. Then for any sub-image, the electronic device determines the coordinate information of each image block in the first image, maps the coordinate information of each image block based on the mapping parameter, and obtains the position feature of each image block. The first image feature of the sub-image includes the image feature of each image block, and the electronic device fuses the image feature and the position feature of each image block in the sub-image to obtain the image feature of each image block, and fuses the image features of the plurality of image blocks in the sub-image to obtain the second image feature of the sub-image.

[0111] In some embodiments, the coordinate information of the sub-image in the first image refers to the coordinate information of the center point of the sub-image in the first image.

[0112] 304, the first image is reduced to a second intermediate image, and the resolution of the second intermediate image is equal to the target resolution.

[0113] In the embodiments of the present application, in addition to encoding the sub-image, the first image can also be reduced to a second intermediate image with a target resolution, and then the second intermediate image is encoded, which is beneficial to retaining the global visual context information of the first image and improving the accuracy of subsequent processing based on image features.

[0114] 305, the second intermediate image is encoded by the image encoding network to obtain the image feature of the second intermediate image.

[0115] Since the resolution of the second intermediate image is equal to the target resolution, the image encoding network can process the second intermediate image.

[0116] The electronic device inputs the second intermediate image into the image encoding network, encodes the second intermediate image by the image encoding network, and outputs the image feature of the second intermediate image.

[0117] 306, the second image features of the plurality of sub-images and the image feature of the second intermediate image are processed by the task processing model to obtain a third image feature.

[0118] After obtaining the second image features of each sub-image and the image features of the second intermediate image, the electronic device inputs the second image features of the plurality of sub-images and the image features of the second intermediate image into the task processing model, and processes the second image features of the plurality of sub-images and the image features of the second intermediate image by the task processing model, for example, deep feature extraction, feature cross, feature mapping, etc., and the task processing model outputs third image features.

[0119] In some embodiments, the task processing model is an LLM (Large Language Model) or a multi-modal model, and the type of the task processing model is not limited in the embodiments of the present application.

[0120] 307. Decoding the third image features by the image coding and decoding model to obtain the second image.

[0121] In some embodiments, the image coding and decoding model includes an image decoding network, and the electronic device decodes the third image features by the image coding and decoding model to obtain the second image, including: converting the first image by the image decoding network to obtain query features; converting the third image features by the image decoding network to obtain key features and value features; fusing the query features, the key features and the value features to obtain fused features; and decoding the fused features by the image decoding network to obtain the second image.

[0122] In some embodiments, the image decoding network includes at least one cross-attention layer, for example, the image decoding network includes 4 cross-attention layers. The electronic device inputs the first image and the noise vector into the image decoding network, and the image decoding network converts the features of the first image and the noise vector, and the converted features are used as query features in the attention mechanism. The electronic device inputs the third image features into the image decoding network, and the image decoding network converts the third image features, and the converted features are used as key features and value features in the attention mechanism. In some embodiments, the image decoding network includes a conversion matrix, which is used to perform the above-mentioned feature conversion, i.e., converting the input features into query features, key features and value features in the attention mechanism, respectively.

[0123] In the embodiments of the present application, when decoding the third image features, the first image is introduced as a reference by steps 304-306, so that the process of decoding the third image features into the second image can learn the fine-grained information in the first image, avoid losing the original content in the first image, and be beneficial to improving the quality and effect of the generated second image, and realize fine-grained image generation.

[0124] And if the reference image is introduced, that is, the method described in FIG. 3A is adopted, the image coding and decoding model can decode the fine-grained image; if the reference image is not introduced, that is, the method described in FIG. 2 is adopted, the image coding and decoding model can decode the coarse-grained image, which improves the flexibility and realizes the generation of multi-granularity images.

[0125] In some embodiments, the third image feature can be uniformly pooled, and the pooled third image feature is decoded through the image decoding network to obtain the second image.

[0126] It should be noted that the embodiments of the present application take the image coding and decoding model and the task processing model as two different models as an example for illustration. In another embodiment, the image coding and decoding model and the task processing model can also be regarded as different parts of an integral model. In some embodiments, the integral model is referred to as a data processing model, as shown in FIG. 3B, the data processing model 310 includes an image coding network 311, a task processing model 312, and an image decoding network 313, wherein the image coding network 311 and the image decoding network 313 constitute an image coding and decoding model, and the image coding network 311 and the image decoding network 313 are connected with the task processing model 312 respectively.

[0127] Specifically, taking the method shown in FIG. 2 as an example, the plurality of sub-images of the first image are input into the image coding network 311, and the first image feature is output; then, the first image feature is fused with the first position feature to obtain the second image feature as the input of the task processing model 312. Then, the third image feature is output via the task processing model 312, and then input into the image decoding network 313, and finally the second image is output.

[0128] On the basis of the above-mentioned embodiments, the target task can also be a multi-modal task, and the data to be processed by the target task includes non-image data in addition to the image, and the detailed process of the data processing method is described in the following embodiment shown in FIG. 4. FIG. 4 is a flowchart of another data processing method provided by an embodiment of the present application, which is executed by an electronic device, which can be the terminal 101 or the server 102 in FIG. 1. Referring to FIG. 4, the method includes:

[0129] 401, obtaining a first image and first non-image data, the first image and the first non-image data are both data to be processed in a target task.

[0130] Wherein, the target task is a multi-modal task, which means a task of processing multiple types of data, and the first non-image data refers to other data in addition to the image, for example, the first non-image data can be text, video or audio, etc., which is not limited by the embodiments of the present application.

[0131] Taking a target task as an example of a multi-modal task combining images and texts, the first non-image data is a first text. The first image and the first text can be an image-text pair, structured image-text data, interleaved image-text data, OCR (Optical Character Recognition) data, and the like. The image-text pair refers to an image and a text describing the content of the image. The structured image-text data refers to an image and a text that structurally describes the image. The interleaved image-text data refers to an image and a text that are interleaved with each other. The OCR data refers to an image and a text obtained by recognizing characters in the image.

[0132] FIG. 5 is a schematic diagram of a multi-modal data sequence according to an embodiment of the present application. As shown in FIG. 5, the data to be processed in a multi-modal task is a multi-modal data sequence, which includes at least one image and at least one text, for example, the multi-modal data sequence includes an image 501, a text 502, and an image 503.

[0133] 402. The first image is divided to obtain a plurality of sub-images, and a first position feature of each sub-image in the first image is obtained, wherein the resolution of each sub-image is equal to a target resolution of an image coding and decoding model that can be processed by the image coding and decoding model after training.

[0134] 403. For each sub-image, the image coding and decoding model is used to encode the sub-image to obtain a first image feature of the sub-image, and the first image feature and the first position feature are fused to obtain a second image feature of the sub-image.

[0135] The process of steps 402-403 is the same as that of steps 302-303 described above, and will not be described here again.

[0136] 404. The first non-image data is subjected to feature extraction to obtain a first data feature.

[0137] In some embodiments, the electronic device subjects the first non-image data to feature extraction by a non-image coding and decoding model to obtain a first data feature. The non-image coding and decoding model is used to process non-image data. For example, when the first non-image data is a text, the non-image coding and decoding model is a text coding and decoding model. When the first non-image data is audio, the non-image coding and decoding model is an audio coding and decoding model. In some embodiments, the non-image coding and decoding model includes a non-image encoding network and a non-image decoding network, and the electronic device subjects the first non-image data to feature extraction by the non-image encoding network to obtain the first data feature.

[0138] In some embodiments, the first non-image data is first text, and the first data feature is a text feature, which can be an embedding vector or the like. In some embodiments, the electronic device maps each word in the first text to its corresponding embedding vector by querying a dictionary that includes a correspondence between a plurality of words and their embedding vectors.

[0139] 405. The second image features of the plurality of sub-images and the first data features of the first non-image data are processed by the task processing model to obtain third image features and second data features.

[0140] After obtaining the second image features of each sub-image and the first data features of the first non-image data, the electronic device inputs the second image features of the plurality of sub-images and the first data features to the task processing model, processes the input features by the task processing model, for example, performs deep feature extraction, feature cross, feature mapping, etc., and outputs third image features and second data features, wherein the third image features are the features of the second image generated by the target task, and the second data features are the features of the non-image data generated by the target task.

[0141] In some embodiments, the first non-image data is text, and the text includes coordinate information of an object in the first image. The electronic device converts the coordinate information into a second position feature, fuses the first data feature with the second position feature to obtain a third data feature, and processes the second image features of the plurality of sub-images and the third data features by the task processing model to obtain third image features and second data features.

[0142] In the embodiments of the present application, in order to improve the processing effect, in the case where the text includes coordinate information of an object in the first image, the coordinate information is converted into a second position feature, which can represent the position of the object in the image. In some embodiments, the electronic device obtains a plurality of candidate position markers, each of which indicates a different position in the image, and determines a target position marker from the plurality of candidate position markers based on the coordinate information, wherein the position in the image indicated by the target position marker is the same as the position in the image indicated by the coordinate information. The electronic device takes the target position marker as the second position feature and adds the second position feature to the first data feature.

[0143] In some embodiments, the electronic device adds an image start tag in front of the second image features of the plurality of sub-images and an image end tag behind the second image features of the plurality of sub-images before inputting the first data features of the first non-image data and the second image features of the plurality of sub-images into the task processing model, so as to distinguish the image features and the data features. For example, the image start tag is "IMG" and the image end tag is " / IMG".

[0144] 406. Decoding the third image features by the image coding and decoding model to obtain the second image.

[0145] The process of this step 406 is the same as that of the above step 307, and will not be repeated here.

[0146] 407. Decoding the second data features to obtain the second non-image data.

[0147] In some embodiments, the electronic device decodes the second data features by a non-image coding and decoding model to obtain the second non-image data. The non-image coding and decoding model is used to process non-image data. In some embodiments, the non-image coding and decoding model includes a non-image encoding network and a non-image decoding network, and the electronic device decodes the second data features by the non-image decoding network to obtain the second non-image data.

[0148] In some embodiments, the first non-image data is a first text, the second non-image data is a second text, and the second data features are text features, which can be embedding vectors, etc. In some embodiments, the electronic device maps the text features to their corresponding words by querying a dictionary to obtain the second text, and the dictionary includes the corresponding relationship between a plurality of words and their embedding vectors.

[0149] Based on the above embodiments, the task processing model can process the features of the multi-modal data, improve the processing capability of the task processing model, and make the task processing model applicable to various types of data in various scenarios, further improving the adaptability of the data processing method.

[0150] FIG. 6 is a schematic diagram of a data processing method provided in an embodiment of the present application. As shown in FIG. 6, the data to be processed in the target task includes a first image and a first text. The first image is divided and scaled to obtain an image sequence 601, which includes divided sub-images 6012-6015 and a second intermediate image 6011 obtained by scaling. The first text is "dog loves playing in the park (or English Lucky likes playing in the park)". Each image 6011-6015 in the image sequence 601 is encoded by an image encoding network (not shown) in an image coding and decoding model to obtain image features 602. The first text is encoded to obtain text features 603. The electronic device also randomly generates image features 604, adds an image start tag "IMG" in front of the image features 602 and 604, adds an image end tag " / IMG" behind the image features 602 and 604, and adds a start tag "BOI" in front of the entire input features.

[0151] As shown in FIG. 6, the electronic device inputs the above-mentioned features and tags into a task processing model, and the task processing model outputs predicted image features 605 and text features 606. The electronic device inputs the image features 605 into an image decoding network in the image coding and decoding model, and the image decoding network outputs a second image 607. The electronic device decodes the text features 606 and can also obtain a second text. The second image 607 and the second text are the processing results of processing the first image and the first text according to the target task.

[0152] FIG. 7 is a schematic diagram of an interactive method based on artificial intelligence provided in an embodiment of the present application. The functions of the AI assistant can be realized through the above-mentioned image coding and decoding model and the task processing model, and the user can interact with the AI assistant. As shown in FIG. 7, in a target task 701, the user inputs an image 711 and a text "show me the photo of the room decoration", and the AI assistant outputs an image 721 and a text "here is the photo provided for you". In a target task 702, the user inputs an image 712 and a text "can you draw a colorful animation character according to the sketch", and the AI assistant outputs an image 722 and a text "here is the animation character created for you". In a target task 703, the user inputs an image 713 and a text "can you create an illustration for this novel", and the AI assistant outputs an image 723 and a text "of course, please enjoy the illustration". In a target task 704, the user inputs an image 714 and a text "can you turn this photo into a watercolor painting", and the AI assistant outputs an image 724 and a text "of course".

[0153] The training process of the task processing model used in the above embodiments can refer to the embodiment shown in FIG. 8. FIG. 8 is a flowchart of a model training method provided by an embodiment of the present application. The embodiment of the present application is executed by an electronic device, which can be the terminal 101 or the server 102 in FIG. 1. Referring to FIG. 8, the method comprises the following steps.

[0154] 801. Obtain a first sample image.

[0155] The first sample image is a sample image required to be processed in the target task. The process of this step 801 is the same as that of the above step 301, and will not be repeated here.

[0156] 802. Process the first sample image based on the target task to obtain a target sample image, and obtain a target sample image feature of the target sample image.

[0157] The target sample image refers to the result of processing the first sample image according to the target task. For example, if the target task is to migrate the style of the first sample image to an animation style, the target sample image refers to the image obtained by migrating the style of the first sample image to an animation style. For example, if the target task is to remove a certain object in the first sample image, the target sample image refers to the image obtained by removing the object in the first sample image.

[0158] In some embodiments, the electronic device obtains the target sample image feature of the target sample image, comprising:

[0159] Divide the target sample image to obtain a plurality of target sub-images, and obtain a second sample position feature of each target sub-image in the target sample image, the resolution of each target sub-image being equal to the target resolution;

[0160] For each target sub-image,

[0161] Encode the target sub-image through the image coding model to obtain an image feature of the target sub-image;

[0162] Fuse the image feature of the target sub-image and the second sample position feature to obtain the target sample image feature of the target sub-image.

[0163] The process of obtaining the target sample image feature of each target sub-image in the target sample image by the electronic device is the same as the process of obtaining the second image feature of each sub-image in the first image in the above embodiments, and will not be repeated here.

[0164] 803. Divide the first sample image to obtain multiple first sample sub-images, and obtain a first sample position feature of each first sample sub-image in the first sample image, wherein the resolution of each first sample sub-image is equal to the target resolution of the image that can be processed by the trained image encoding and decoding model.

[0165] In some embodiments, the training process of the image coding and decoding model is decoupled from the training process of the task processing model. The embodiment shown in Figure 8 is the process of training the task processing model. In this embodiment, the image coding and decoding model refers to the trained model. The training process of the image coding and decoding model is shown in the embodiment shown in Figure 11 below and is not explained here.

[0166] 804. For each first sample sub-image, encode the first sample sub-image using an image encoding and decoding model to obtain a first sample image feature of the first sample sub-image, and fuse the first sample image feature with the first sample position feature to obtain a second sample image feature of the first sample sub-image.

[0167] The process of step 803 to step 804 is the same as the process of step 302 to step 303 above, and will not be repeated here.

[0168] 805. Process the second sample image features of the plurality of first sample sub-images using the initial processing model to obtain third sample image features.

[0169] The initial processing model is used to process features, such as deep feature extraction, feature intersection, feature mapping, etc.

[0170] In some embodiments, the initial processing model is a pre-trained model. The pre-training process of the initial processing model is described in the embodiment shown in FIG14 below and will not be described here.

[0171] 806. Based on the third sample image features and the target sample image features, the initial processing model is trained to obtain a task processing model for processing the target task.

[0172] In the embodiments of the present application, the target sample image features are the result of processing the first sample image according to the target task, and the goal of the initial processing model is to obtain the target sample image features. That is, the closer the third sample image features generated by the initial processing model are to the target sample image features, the more accurate the training process of the initial processing model. Therefore, the electronic device trains the initial processing model based on the third sample image features and the target sample image features to obtain a task processing model, so that the image features output by the task processing model tend to the target sample image features.

[0173] In some embodiments, the electronic device determines a loss parameter based on the third sample image feature and the target sample image feature, trains the initial processing model based on the loss parameter to obtain the task processing model. The loss parameter can be cross-entropy loss or mean square error loss, etc., which is not limited in the embodiments of the present application.

[0174] The method provided by the embodiments of the present application divides the image into a plurality of sub-images with a resolution equal to the target resolution in the case that the image to be processed in the target task does not match the target resolution corresponding to the image coding and decoding model, so as to directly encode the sub-images by the image coding and decoding model. Moreover, the position feature of the sub-image is fused into the image feature of the sub-image, so as to retain the position information of each sub-image in the complete image. Furthermore, the image feature of each sub-image is processed by the initial processing model, and the model is trained based on the processed image feature and the image feature to be generated, so as to obtain the task processing model for processing the target task. Therefore, by dividing the image and retaining the position information, the task processing model can be applied to images with any resolution, and the adaptability of the data processing method is improved.

[0175] FIG. 9 is a flowchart of a method for training a task processing model according to an embodiment of the present application. The method is executed by an electronic device, which can be the terminal 101 or the server 102 in FIG. 1. Referring to FIG. 9, the method includes the following steps.

[0176] 901. Obtain a first sample image and first sample non-image data, both of which are data to be processed in a target task.

[0177] The target task is a multi-modal task, which means a task of processing multiple types of data. The first sample non-image data refers to data other than images, such as text, video, audio, etc., which is not limited in the embodiments of the present application.

[0178] The process of step 901 is the same as that of step 401 described above, which is not repeated here.

[0179] 902. Process the first sample image based on the target task to obtain a target sample image, and obtain a target sample image feature of the target sample image.

[0180] The target sample image is the result of processing the first sample image according to the target task.

[0181] 903. Divide the first sample image to obtain multiple first sample sub-images, and obtain a first sample position feature of each first sample sub-image in the first sample image, wherein the resolution of each first sample sub-image is equal to the target resolution of the image that can be processed by the trained image encoding and decoding model.

[0182] 904. For each first sample sub-image, encode the first sample sub-image using an image encoding and decoding model to obtain a first sample image feature of the first sample sub-image, and fuse the first sample image feature with the first sample position feature to obtain a second sample image feature of the first sample sub-image.

[0183] The process of step 903 to step 904 is the same as the process of step 803 to step 804 above, and will not be repeated here.

[0184] 905. Perform feature extraction on the first sample non-image data to obtain first sample data features.

[0185] The process of step 905 is the same as that of step 404 above, and will not be repeated here.

[0186] 906. Process the second sample image features and the first sample data features of the plurality of first sample sub-images using the initial processing model to obtain third sample image features and second sample data features.

[0187] In some embodiments, when the electronic device predicts the second sample data feature through the initial processing model, it obtains it in the manner of predicting the subsequent feature according to the preceding feature. For example, the first sample non-image data is the first sample text, and the first sample data feature is the first sample text feature, which includes the first sample text feature of each sample word in the first sample text. The electronic device sequentially inputs the second sample image features of multiple first sub-images and the first sample text features of multiple sample words into the initial processing model. The initial processing model determines the second sample text feature of the first predicted word based on the feature preceding the first sample text feature of the first sample word, and determines the second sample text feature of the second predicted word based on the feature preceding the first sample text feature of the second sample word, and so on, until the second sample text features of all predicted words are output to constitute the second sample data feature. If the initial processing model is sufficiently accurate, then the second sample text features of the multiple predicted words predicted by the initial processing model should tend to the first sample text features of the multiple sample words.

[0188] 907. Based on the third sample image feature, the target sample image feature, the second sample data feature, and the first sample data feature, the initial processing model is trained to obtain a task processing model.

[0189] In the embodiments of the present application, the target of the initial processing model is to generate the target sample image feature and the first sample data feature, that is, the closer the third sample image feature generated by the initial processing model to the target sample image feature and the closer the second sample data feature generated by the initial processing model to the first sample data feature, the more accurate the model is.

[0190] Therefore, the electronic device trains the initial processing model based on the third sample image feature and the target sample image feature and the second sample data feature and the first sample data feature to obtain the task processing model, so that the image feature generated by the task processing model tends to be the target sample image feature and the data feature generated by the task processing model tends to be the first sample data feature.

[0191] In some embodiments, the electronic device determines a first loss parameter based on the third sample image feature and the target sample image feature, determines a second loss parameter based on the second sample data feature and the first sample data feature, fuses the first loss parameter and the second loss parameter to obtain a fused loss parameter, and trains the initial processing model based on the fused loss parameter to obtain the task processing model. The above loss parameters can be cross-entropy loss or mean square error loss, etc., which are not limited in the embodiments of the present application.

[0192] FIG. 10 is a flowchart of another method for training a task processing model according to an embodiment of the present application. As shown in FIG. 10, the data to be processed in the target task includes a first sample image and a first sample text. The first sample image is divided and scaled to obtain an image sequence 1001, which includes divided sub-images 10012-10015 and a scaled image 10011. The first sample text is “dog loves playing in the park (or English Lucky likes playing in the park)”. Each image 10011-10015 in the image sequence 1001 is encoded by an image encoding network (not shown) in the image coding model to obtain an image feature 1002. The first text is encoded to obtain a text feature 1003. The electronic device also randomly generates an image feature 1004, adds an image start tag “IMG” in front of the image feature 1002 and the image feature 1004, adds an image end tag “ / IMG” behind the image feature 1002 and the image feature 1004, and adds a start tag “BOI” in front of the entire input feature.

[0193] As shown in FIG. 10, the electronic device inputs the above-mentioned respective features and labels into an initial processing model, and the initial processing model outputs predicted image features 1005 and text features 1006. The electronic device further obtains image features 1007 of a target sample image, determines a first loss parameter based on the image features 1005 and the image features 1007, determines a second loss parameter based on the text features 1006 and the text features 1003, and trains the initial processing model based on the first loss parameter and the second loss parameter to obtain a task processing model.

[0194] FIG. 11 is a flowchart of a method for training an image coding model according to an embodiment of the present application. The method is performed by an electronic device, which can be the terminal 101 or the server 102 in FIG. 1. As shown in FIG. 11, the method includes the following steps.

[0195] 1101. Obtain a second sample image.

[0196] The second sample image can be an image of any field, any type, and any resolution, and the present application does not limit the second sample image.

[0197] 1102. Divide the second sample image to obtain a plurality of second sample sub-images, and obtain a third sample position feature of each second sample sub-image in the second sample image, wherein the resolution of each second sample sub-image is equal to the target resolution.

[0198] 1103. For each second sample sub-image, encode the second sample sub-image by using an initial image coding model to obtain a fourth sample image feature of the second sample sub-image, and fuse the fourth sample image feature and the third sample position feature to obtain a fifth sample image feature of the second sample sub-image.

[0199] In some embodiments, the electronic device obtains the third sample position feature of each second sample sub-image in the second sample image, including:

[0200] For any second sample sub-image,

[0201] determine coordinate information of the second sample sub-image in the second sample image;

[0202] map the coordinate information based on a preset mapping parameter to obtain the third sample position feature of the second sample sub-image.

[0203] Subsequently, in step 1105, the initial image coding model and the preset mapping parameter are trained based on the difference between the third sample image and the second sample image.

[0204] The initial image coding model includes an initial image encoding network and an initial image decoding network. The initial image encoding network can be an initialized VAE or ViT. The initial image decoding network includes a plurality of initialized cross-attention layers.

[0205] The preset mapping parameters refer to initial values of the left mapping parameter, the right mapping parameter, the upper mapping parameter, and the lower mapping parameter.

[0206] The processes of steps 1102-1103 are the same as those of steps 302-303, and thus are not described herein again.

[0207] 1104. Decoding, by the initial image coding model, the fifth sample image features of the plurality of second sample sub-images to obtain a third sample image.

[0208] In some embodiments, the electronic device obtains a reference sample image, the reference sample image has a part of content that is the same as the second sample image, that is, the two have the same content and different content; converts, by the initial image coding model, a feature of the reference sample image to obtain a sample query feature; converts, by the initial image coding model, a feature of the fifth sample image of the plurality of second sample sub-images to obtain a sample key feature and a sample value feature; fuses the sample query feature, the sample key feature, and the sample value feature to obtain a sample fusion feature; and decodes, by the initial image coding model, the sample fusion feature to obtain the third sample image.

[0209] For example, the second sample image includes two dogs, and the reference sample image is an image obtained by adding a third dog to the second sample image. Or, the second sample image includes three people, and the reference sample image is an image obtained by adding a bunch of fresh flowers to the second sample image.

[0210] 1105. Training, based on the third sample image and the second sample image, the initial image coding model to obtain a trained image coding model.

[0211] In the embodiments of the present application, the training target of the image coding model is to reconstruct the second sample image, that is, the closer the third sample image generated by the image coding model to the second sample image, the more accurate the model. Therefore, the electronic device trains the initial image coding model based on the third sample image and the second sample image, so that the image generated by the trained image coding model tends to be the second sample image.

[0212] In some embodiments, the electronic device determines a loss parameter based on the third sample image and the second sample image, trains the initial image coding model based on the loss parameter to obtain the trained image coding model. The loss parameter can be cross-entropy loss or mean square error loss, etc., which is not limited in the embodiments of the present application.

[0213] It should be noted that the training process of the image coding model can be divided into two stages. In the first stage, the image features are directly decoded by the initial image coding model without introducing the reference image during decoding. In the second stage, the image features are decoded based on the reference image by the initial image coding model during decoding.

[0214] FIG. 12 is a schematic diagram of a training method of an image coding model according to an embodiment of the present application. As shown in FIG. 12, the initial image coding model includes an image encoding network 1210 and an image decoding network 1220.

[0215] In the first stage, the original image A is encoded and decoded by the image encoding network 1210 and the image decoding network 1220 respectively, and finally the reconstructed image B is obtained. Then, the image encoding network 1210 and the image decoding network 1220 are trained based on the difference between the reconstructed image B and the original image A. At this time, the difference mainly refers to the difference in semantic consistency. In the image decoding network 1220, in addition to the input image features, noise is also input.

[0216] In the second stage, the original image C1 is encoded and decoded by the image encoding network 1210 and the image decoding network 1220 respectively, and at the same time the reference image C2 is introduced during decoding, and finally the reconstructed image D is obtained. Then, the image encoding network 1210 and the image decoding network 1220 are trained based on the difference between the reconstructed image D and the original image C1. At this time, the difference refers to the difference in fine-grained semantic consistency. In the image decoding network 1220, in addition to the input image features, the reference image C2 and the noise are also input.

[0217] FIG. 13 is a comparison diagram of reconstruction results of an image coding model according to an embodiment of the present application. As shown in FIG. 13, the original image A1 is reconstructed to obtain the reconstructed image B1 by using the related art, the original image A1 is reconstructed to obtain the reconstructed image B2 by using the method of the present application without introducing the reference image, and the original image A1 is reconstructed to obtain the reconstructed image B3 by using the method of the present application with the introduction of the reference image A2. As can be seen from FIG. 13, the image B3 reconstructed by using the method of the present application with the introduction of the reference image is closer to the original image A1, and the reconstruction effect is better.

[0218] The embodiments of FIGS. 8 and 9 above are trained on the basis of an initial processing model to obtain a task processing model, thereby obtaining a task processing model specially used for executing a specific target task. The initial processing model can be a pre-trained model, and the pre-training process of the initial processing model is the same as the training process of the task processing model, except that the training samples used in the training process of the task processing model are data under the target task, and the training samples used in the pre-training process of the initial processing model can be data under any scenario. The pre-training process of the initial processing model is described in the embodiment of FIG. 14 below.

[0219] FIG. 14 is a flowchart of a pre-training method of an initial processing model according to an embodiment of the present application. The embodiment of the present application is executed by an electronic device, which can be the terminal 101 or the server 102 in FIG. 1. Referring to FIG. 14, the method includes the following steps.

[0220] 1401, obtaining a fourth sample image and obtaining a sample image feature of a fifth sample image, the fifth sample image being an image associated with the fourth sample image in content.

[0221] The fifth sample image is an associated image of the fourth sample image, for example, the fourth sample image and the fifth sample image belong to the same image-text data, and the fifth sample image is the context of the fourth sample image. Or, the similarity between the fifth sample image and the fourth sample image reaches a preset threshold, and the present embodiment is not limited thereto.

[0222] The process of obtaining the sample image feature of the fifth sample image by the electronic device is the same as the process of obtaining the target sample image feature of the target sample image in the above steps, and will not be repeated here.

[0223] 1402, dividing the fourth sample image to obtain a plurality of fourth sample sub-images, and obtaining a fourth sample position feature of each fourth sample sub-image in the fourth sample image, wherein the resolution of each fourth sample sub-image is equal to the target resolution.

[0224] 1403, for each fourth sample sub-image, encoding the fourth sample sub-image by using the image coding and decoding model to obtain a sixth sample image feature of the fourth sample sub-image, and fusing the sixth sample image feature and the fourth sample position feature to obtain a seventh sample image feature of the fourth sample sub-image.

[0225] 1404, processing the seventh sample image features of the plurality of fourth sample sub-images by using the initialized initial processing model to obtain an eighth sample image feature.

[0226] 1405、based on the eighth sample image feature and the sample image feature of the fifth sample image, pre-training the initial processing model.

[0227] The processes of steps 1402-1405 are the same as those of steps 803-806 described above, and are not described here again.

[0228] FIG. 15 is a comparison diagram of a data processing result provided by an embodiment of the present application, and the target task is to process an image based on the requirement described by a text in the case of giving the text and the image. As shown in FIG. 15, the images generated by processing the text and the image in different target tasks by using the related techniques 1-4 and the method of the present application are given respectively. As can be seen from FIG. 15, compared with each related technique, the image generated by using the method of the present application is more in line with the processing requirement described by the text in the target task, and the present application can improve the adaptability of the data processing method and also improve the effect of the data.

[0229] FIG. 16 is a structural schematic diagram of a data processing apparatus provided by an embodiment of the present application. Referring to FIG. 16, the data processing apparatus 1600 includes:

[0230] The acquisition module 1601 is configured to acquire a first image to be processed in a target task.

[0231] The image processing module 1602 is configured to divide the first image to obtain a plurality of sub-images, and acquire a first position feature of each sub-image in the first image, wherein the resolution of each sub-image is equal to a target resolution of an image that can be processed by the trained image coding and decoding model.

[0232] The encoding module 1603 is configured to, for each sub-image, encode the sub-image by using the image coding and decoding model to obtain a first image feature of the sub-image, and fuse the first image feature and the first position feature to obtain a second image feature of the sub-image.

[0233] The feature processing module 1604 is configured to process the second image features of the plurality of sub-images by using the trained task processing model to obtain a third image feature.

[0234] The decoding module 1605 is configured to decode the third image feature by using the image coding and decoding model to obtain a second image.

[0235] In some embodiments, referring to FIG. 16, the image processing module 1602 is configured to:

[0236] Determining multiple division strategies, the division strategies including a first number of divisions, a second number of divisions, and a division ratio, wherein the first number of divisions refers to the number of divisions of the image in the length direction, the second number of divisions refers to the number of divisions of the image in the width direction, and the division ratio refers to the ratio between the first number of divisions and the second number of divisions;

[0237] Determining a target segmentation strategy from the multiple segmentation strategies based on the aspect ratio of the first image, wherein the segmentation ratio in the target segmentation strategy is closest to the aspect ratio;

[0238] Based on the target segmentation strategy and the target resolution, the first image is segmented to obtain the multiple sub-images.

[0239] In some embodiments, referring to FIG. 16 , the image processing module 1602 is configured to:

[0240] generating a target size based on the target partitioning strategy and the target resolution;

[0241] Upsampling the first image based on the target size to obtain a first intermediate image;

[0242] The first intermediate image is divided according to the target division strategy to obtain the multiple sub-images.

[0243] In some embodiments, referring to FIG. 16 , the image processing module 1602 is configured to:

[0244] For any sub-image,

[0245] Determining coordinate information of the sub-image in the first image;

[0246] The coordinate information is mapped based on mapping parameters to obtain a first position feature of the sub-image, wherein the mapping parameters are obtained through training.

[0247] In some embodiments, referring to FIG16 , the image coding model includes an image coding network, an image processing module 1602 , and is further configured to reduce the first image to a second intermediate image, wherein the resolution of the second intermediate image is equal to the target resolution;

[0248] The encoding module 1603 is further configured to encode the second intermediate image through the image encoding network to obtain image features of the second intermediate image;

[0249] The feature processing module 1604 is configured to process the second image features of the plurality of sub-images and the image features of the second intermediate image using the task processing model to obtain the third image features.

[0250] In some embodiments, referring to FIG. 16, the image coding model includes an image decoding network, a decoding module 1606, configured to:

[0251] perform feature conversion on the first image through the image decoding network to obtain query features;

[0252] perform feature conversion on the third image features through the image decoding network to obtain key features and value features;

[0253] fuse the query features, the key features, and the value features to obtain fused features;

[0254] decode the fused features through the image decoding model to obtain the second image.

[0255] In some embodiments, referring to FIG. 16, the target task is a multi-modal task, and the data to be processed in the multi-modal task further includes first non-image data;

[0256] a feature processing module 1604, configured to process the second image features of the plurality of sub-images and first data features of the first non-image data through the task processing model to obtain third image features and second data features.

[0257] In some embodiments, referring to FIG. 16, the decoding module 1606 is further configured to decode the second data features to obtain second non-image data.

[0258] In some embodiments, referring to FIG. 16, the first non-image data is text, and the text includes coordinate information of an object in the first image; the feature processing module 1604 is configured to:

[0259] convert the coordinate information into second position features, fuse the first data features and the second position features to obtain third data features;

[0260] process the second image features of the plurality of sub-images and the third data features through the task processing model to obtain the third image features and the second data features.

[0261] It should be noted that the data processing apparatus provided in the above embodiments is only exemplified by the division of the above functional modules, and in actual application, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the electronic device is divided into different functional modules to complete all or part of the functions described above. In addition, the data processing apparatus and the data processing method provided in the above embodiments belong to the same concept, and the specific implementation process is detailed in the method embodiments, which will not be repeated here.

[0262] FIG. 17 is a structural schematic diagram of a model training apparatus provided by an embodiment of the present application. Referring to FIG. 17, the model training apparatus 1700 comprises:

[0263] The acquisition module 1701 is configured to acquire a first sample image; perform processing on the first sample image based on a target task to obtain a target sample image, and acquire a target sample image feature of the target sample image;

[0264] The image processing module 1702 is configured to divide the first sample image to obtain a plurality of first sample sub-images, and acquire a first sample position feature of each first sample sub-image in the first sample image, wherein a resolution of each first sample sub-image is equal to a target resolution of an image coding and decoding model capable of being processed by a trained image.

[0265] The encoding module 1703 is configured to, for each first sample sub-image, encode the first sample sub-image by using the image coding and decoding model to obtain a first sample image feature of the first sample sub-image, and fuse the first sample image feature and the first sample position feature to obtain a second sample image feature of the first sample sub-image.

[0266] The feature processing module 1704 is configured to process the second sample image features of the plurality of first sample sub-images by using an initial processing model to obtain third sample image features; and

[0267] The first training module 1705 is configured to train the initial processing model based on the third sample image features and the target sample image feature to obtain a task processing model for processing the target task.

[0268] In some embodiments, referring to FIG. 18, in the model training apparatus 1800, the acquisition module 1701 is configured to:

[0269] divide the target sample image to obtain a plurality of target sub-images, and acquire a second sample position feature of each target sub-image in the target sample image, wherein a resolution of each target sub-image is equal to the target resolution;

[0270] for each target sub-image,

[0271] encode the target sub-image by using the image coding and decoding model to obtain an image feature of the target sub-image;

[0272] fuse the image feature of the target sub-image and the second sample position feature to obtain the target sample image feature of the target sub-image.

[0273] In some embodiments, referring to FIG. 18,

[0274] The acquisition module 1701 is further configured to acquire a second sample image.

[0275] The image processing module 1702 is further configured to divide the second sample image to obtain a plurality of second sample sub-images, and acquire a third sample position feature of each second sample sub-image in the second sample image, where a resolution of each second sample sub-image is equal to the target resolution.

[0276] The encoding module 1703 is further configured to, for each second sample sub-image, encode the second sample sub-image by using an initial image coding and decoding model to obtain a fourth sample image feature of the second sample sub-image, and fuse the fourth sample image feature and the third sample position feature to obtain a fifth sample image feature of the second sample sub-image.

[0277] The model training apparatus 1800 further includes:

[0278] The decoding module 1706 is configured to decode the fifth sample image features of the plurality of second sample sub-images by using the initial image coding and decoding model to obtain a third sample image.

[0279] The second training module 1707 is configured to train the initial image coding and decoding model based on the third sample image and the second sample image to obtain a trained image coding and decoding model.

[0280] In some embodiments, referring to FIG. 18, the decoding module 1706 is configured to:

[0281] acquire a reference sample image, where the reference sample image has a part of content identical to the second sample image;

[0282] perform feature conversion on the reference sample image by using the initial image coding and decoding model to obtain a sample query feature;

[0283] perform feature conversion on the fifth sample image features of the plurality of second sample sub-images by using the initial image coding and decoding model to obtain a sample key feature and a sample value feature;

[0284] fuse the sample query feature, the sample key feature, and the sample value feature to obtain a sample fusion feature;

[0285] decode the sample fusion feature by using the initial image coding and decoding model to obtain the third sample image.

[0286] In some embodiments, referring to FIG. 18, the obtaining module 1701 is configured to:

[0287] For any second sample sub-image,

[0288] determine coordinate information of the second sample sub-image in the second sample image;

[0289] map the coordinate information based on preset mapping parameters to obtain the third sample position feature of the second sample sub-image;

[0290] The second training module 1707 is configured to:

[0291] train the initial image coding model and the preset mapping parameters based on the difference between the third sample image and the second sample image.

[0292] In some embodiments, referring to FIG. 18, the initial processing model is pre-trained, and the obtaining module 1701 is further configured to obtain a fourth sample image and obtain a sample image feature of a fifth sample image, the fifth sample image being an image associated with the fourth sample image in content;

[0293] The image processing module 1702 is further configured to divide the fourth sample image to obtain a plurality of fourth sample sub-images and obtain a fourth sample position feature of each fourth sample sub-image in the fourth sample image, wherein the resolution of each fourth sample sub-image is equal to the target resolution.

[0294] The encoding module 1703 is further configured to, for each fourth sample sub-image, encode the fourth sample sub-image by using the image coding model to obtain a sixth sample image feature of the fourth sample sub-image, and fuse the sixth sample image feature and the fourth sample position feature to obtain a seventh sample image feature of the fourth sample sub-image.

[0295] The feature processing module 1704 is further configured to process the seventh sample image features of the plurality of fourth sample sub-images by using the initialized initial processing model to obtain an eighth sample image feature.

[0296] The model training apparatus 1800 further includes:

[0297] The third training module 1708 is configured to pre-train the initial processing model based on the eighth sample image feature and the sample image feature of the fifth sample image.

[0298] In some embodiments, referring to FIG. 18, the target task is a multi-modal task, the data to be processed in the multi-modal task includes the first sample image and first sample non-image data; the feature processing module 1704 is configured to:

[0299] perform processing on the second sample image features of the plurality of first sample sub-images and the first sample data features of the first sample non-image data by using the initial processing model, to obtain third sample image features and second sample data features;

[0300] The first training module 1705 is configured to:

[0301] train the initial processing model based on the third sample image features, the target sample image features, the second sample data features and the first sample data features, to obtain the task processing model.

[0302] It should be noted that the model training apparatus provided in the above embodiments is only used as an example for the division of the above functional modules. In actual applications, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the electronic device is divided into different functional modules to complete all or part of the above-described functions. In addition, the model training apparatus and the model training method provided in the above embodiments belong to the same concept, and the specific implementation process is detailed in the method embodiments, which will not be described here.

[0303] The embodiments of the present application further provide an electronic device, which includes a processor and a memory. The memory stores at least one computer program, which is loaded and executed by the processor to implement the operations performed in the data processing method of the above embodiments.

[0304] In some embodiments, the electronic device is provided as a terminal. FIG. 19 shows a structural schematic diagram of a terminal 1900 provided in an example embodiment of the present application.

[0305] The terminal 1900 includes a processor 1901 and a memory 1902.

[0306] The processor 1901 can include one or more processing cores, such as a 4-core processor, an 8-core processor, and the like. The processor 1901 can be implemented in at least one of a hardware form of a DSP (Digital Signal Processing), an FPGA (Field Programmable Gate Array), a PLA (Programmable Logic Array). The processor 1901 can also include a main processor and a coprocessor, the main processor being a processor for processing data in an awake state, also referred to as a CPU (Central Processing Unit), and the coprocessor being a low-power processor for processing data in a standby state. In some embodiments, the processor 1901 can be integrated with a GPU (Graphics Processing Unit) for rendering and drawing content required to be displayed by the display screen. In some embodiments, the processor 1901 can further include an AI (Artificial Intelligence) processor for processing computing operations related to machine learning.

[0307] The memory 1902 can include one or more computer-readable storage media that can be non-transitory. The memory 1902 can also include a high-speed random access memory, and a nonvolatile memory such as one or more disk storage devices, flash storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1902 is used to store at least one computer program for being executed by the processor 1901 to implement the data processing method or model training method provided by the method embodiments in the present application.

[0308] In some embodiments, the terminal 1900 can also optionally include a peripheral device interface 1903 and at least one peripheral device. The processor 1901, the memory 1902, and the peripheral device interface 1903 can be connected through a bus or a signal line. Each peripheral device can be connected to the peripheral device interface 1903 through a bus, a signal line, or a circuit board. In some embodiments, the peripheral device includes at least one of a radio frequency circuit 1904, a display screen 1905, a camera component 1906, an audio circuit 1907, and a power supply 1908.

[0309] The peripheral interface 1903 can be used to connect at least one I / O (Input / Output) related peripheral device to the processor 1901 and the memory 1902. In some embodiments, the processor 1901, the memory 1902 and the peripheral interface 1903 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 1901, the memory 1902 and the peripheral interface 1903 can be implemented on a separate chip or circuit board, and the present embodiments are not limited in this regard.

[0310] The radio frequency circuit 1904 is used to receive and send RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 1904 communicates with a communication network and other communication devices through electromagnetic signals. The radio frequency circuit 1904 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. In some embodiments, the radio frequency circuit 1904 includes an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a subscriber identity module card, and the like. The radio frequency circuit 1904 can communicate with other devices through at least one wireless communication protocol. The wireless communication protocol includes but is not limited to: a metropolitan area network, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 1904 can also include NFC (Near Field Communication) related circuitry, and the present application is not limited in this regard.

[0311] The display screen 1905 is configured to display a UI (User Interface). The UI can include graphics, text, icons, video, and any combination thereof. When the display screen 1905 is a touch display screen, the display screen 1905 is also capable of capturing touch signals on or above the surface of the display screen 1905. The touch signals can be input to the processor 1901 as control signals for processing. At this time, the display screen 1905 can also be configured to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 1905 can be one, disposed on the front panel of the terminal 1900; in other embodiments, the display screen 1905 can be at least two, respectively disposed on different surfaces of the terminal 1900 or in a folding design; in other embodiments, the display screen 1905 can be a flexible display screen, disposed on a curved surface or a folding surface of the terminal 1900. Even, the display screen 1905 can also be disposed in an irregular shape other than a rectangle, i.e., a special-shaped screen. The display screen 1905 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0312] The camera assembly 1906 is configured to capture images or videos. In some embodiments, the camera assembly 1906 includes a front camera and a rear camera. The front camera is disposed on the front panel of the terminal 1900, and the rear camera is disposed on the back of the terminal 1900. In some embodiments, the rear camera is at least two, which is any one of a main camera, a depth-of-field camera, a wide-angle camera, and a telephoto camera, to realize the background blur function of the main camera and the depth-of-field camera, the panorama shooting and VR (Virtual Reality) shooting function of the main camera and the wide-angle camera, or other fusion shooting functions. In some embodiments, the camera assembly 1906 can also include a flash. The flash can be a single-color-temperature flash or a dual-color-temperature flash. The dual-color-temperature flash refers to the combination of a warm light flash and a cold light flash, which can be used for light compensation under different color temperatures.

[0313] The audio circuit 1907 can include a microphone and a speaker. The microphone is used to collect sound waves of a user and an environment, and convert the sound waves into an electrical signal input to the processor 1901 for processing, or input to the radio frequency circuit 1904 to realize voice communication. The microphone can be multiple for the purpose of stereo sound collection or noise reduction, and is arranged at different parts of the terminal 1900. The microphone can also be an array microphone or an omnidirectional collection microphone. The speaker is used to convert an electrical signal from the processor 1901 or the radio frequency circuit 1904 into sound waves. The speaker can be a conventional diaphragm speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, not only can it convert an electrical signal into a sound wave audible to humans, but also can convert an electrical signal into an inaudible sound wave to humans for ranging purposes. In some embodiments, the audio circuit 1907 can also include a headphone jack.

[0314] The power supply 1908 is used to supply power to various components in the terminal 1900. The power supply 1908 can be alternating current, direct current, disposable battery or rechargeable battery. When the power supply 1908 includes a rechargeable battery, the rechargeable battery can support wired charging or wireless charging. The rechargeable battery can also be used to support fast charging technology.

[0315] Those skilled in the art can understand that the structure shown in FIG. 19 does not constitute a limitation on the terminal 1900, and can include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0316] In some embodiments, the electronic device is provided as a server. FIG. 20 is a structural schematic diagram of a server provided by an embodiment of the present application. The server 2000 can have a large difference due to different configurations or performances, and can include one or more processors (Central Processing Units, CPUs) 2001 and one or more memories 2002, wherein the memory 2002 stores at least one computer program, and the at least one computer program is loaded and executed by the processor 2001 to implement the method provided by each method embodiment. Of course, the server can also have a wired or wireless network interface, a keyboard, and an input and output interface, etc. to input and output, and the server can also include other components for realizing the functions of the device, which are not described here.

[0317] The embodiment of the present application also provides a computer readable storage medium, which stores at least one computer program, and the at least one computer program is loaded and executed by a processor to implement the operations performed by the data processing method or model training method of the above embodiments.

[0318] The embodiment of the present application further provides a computer program product comprising a computer program, which is loaded and executed by a processor to implement the operations performed by the data processing method or the model training method of the above embodiment.

[0319] Those skilled in the art can understand that all or part of the steps of the above-mentioned embodiments can be completed by hardware, or can be instructed by a program to complete the related hardware, and the program can be stored in a computer readable storage medium. The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk.

[0320] The above description is only optional embodiments of the present application and does not limit the embodiments of the present application. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the embodiments of the present application shall be included in the protection scope of the present application.

Claims

1. A data processing method, executed by an electronic device, comprising: obtaining a first image to be processed in a target task; dividing the first image to obtain a plurality of sub-images, and obtaining a first position feature of each sub-image in the first image, wherein a resolution of each sub-image is equal to a target resolution of an image coding and decoding model after training; for each sub-image, encoding the sub-image by the image coding and decoding model to obtain a first image feature of the sub-image, and fusing the first image feature and the first position feature to obtain a second image feature of the sub-image; processing the second image features of the plurality of sub-images by a trained task processing model to obtain a third image feature; and decoding the third image feature by the image coding and decoding model to obtain a second image.

2. The method of claim 1, wherein, The dividing the first image to obtain a plurality of sub-images comprises: determining a plurality of division strategies, the division strategies comprising a first division number, a second division number, and a division ratio, the first division number referring to a division number of the image in a length direction, the second division number referring to a division number of the image in a width direction, and the division ratio referring to a ratio between the first division number and the second division number; determining a target division strategy from the plurality of division strategies based on an aspect ratio of the first image, the division ratio in the target division strategy being closest to the aspect ratio; dividing the first image based on the target division strategy and the target resolution to obtain the plurality of sub-images.

3. The method of claim 2, wherein, The dividing the first image based on the target division strategy and the target resolution to obtain the plurality of sub-images comprises: generating a target size based on the target division strategy and the target resolution; up-sampling the first image based on the target size to obtain a first intermediate image; dividing the first intermediate image according to the target division strategy to obtain the plurality of sub-images.

4. The method of any one of claims 1-3, wherein, The obtaining a first position feature of each sub-image in the first image comprises: for any sub-image, determining coordinate information of the sub-image in the first image; mapping the coordinate information based on a mapping parameter to obtain a first position feature of the sub-image, the mapping parameter being obtained by training.

5. The method of any one of claims 1-4, wherein, The image coding and decoding model comprises an image encoding network, and the method further comprises: reducing the first image to a second intermediate image, the resolution of the second intermediate image being equal to the target resolution; encoding the second intermediate image by the image encoding network to obtain an image feature of the second intermediate image; The processing the second image features of the plurality of sub-images by the trained task processing model to obtain a third image feature comprises: processing the second image features of the plurality of sub-images and the image feature of the second intermediate image by the task processing model to obtain the third image feature.

6. The method of any one of claims 1-5, wherein, The image coding model comprises an image decoding network, and the third image feature is decoded by the image coding model to obtain a second image, which comprises: The first image is feature-converted by the image decoding network to obtain a query feature; The third image feature is feature-converted by the image decoding network to obtain a key feature and a value feature; The query feature, the key feature and the value feature are fused to obtain a fused feature; The fused feature is decoded by the image decoding model to obtain the second image.

7. The method of any one of claims 1-6, wherein, The target task is a multi-modal task, and the data to be processed in the multi-modal task further comprises first non-image data; the second image feature of the plurality of sub-images is processed by the trained task processing model to obtain a third image feature, which comprises: The second image feature of the plurality of sub-images and the first data feature of the first non-image data are processed by the task processing model to obtain the third image feature and a second data feature.

8. The method of claim 7, further comprising: decoding the second data feature to obtain second non-image data.

9. The method of claim 7 or 8, wherein, The first non-image data is text, and the text comprises coordinate information of an object in the first image; the second image feature of the plurality of sub-images and the first data feature of the first non-image data are processed by the task processing model to obtain the third image feature and a second data feature, which comprises: The coordinate information is converted into a second position feature, and the first data feature and the second position feature are fused to obtain a third data feature; The second image feature of the plurality of sub-images and the third data feature are processed by the task processing model to obtain the third image feature and the second data feature.

10. A model training method performed by an electronic device, the method comprising: obtaining a first sample image; processing the first sample image based on a target task to obtain a target sample image and obtaining a target sample image feature of the target sample image; dividing the first sample image to obtain a plurality of first sample sub-images and obtaining a first sample position feature of each first sample sub-image in the first sample image, wherein the resolution of each first sample sub-image is equal to a target resolution of an image that can be processed by a trained image coding model; for each first sample sub-image, encoding the first sample sub-image by the image coding model to obtain a first sample image feature of the first sample sub-image, and fusing the first sample image feature and the first sample position feature to obtain a second sample image feature of the first sample sub-image; processing the second sample image features of the plurality of first sample sub-images by an initial processing model to obtain third sample image features; and Based on the third sample image feature and the target sample image feature, the initial processing model is trained to obtain a task processing model for processing the target task.

11. The method of claim 10, wherein, The target sample image feature of the target sample image is obtained by: dividing the target sample image to obtain a plurality of target sub-images, and obtaining a second sample position feature of each target sub-image in the target sample image, wherein a resolution of each target sub-image is equal to the target resolution; for each target sub-image, encoding the target sub-image by the image coding model to obtain an image feature of the target sub-image; fusing the image feature of the target sub-image and the second sample position feature to obtain the target sample image feature of the target sub-image.

12. The method of claim 10 or 11, wherein, The training process of the image coding model includes: obtaining a second sample image; dividing the second sample image to obtain a plurality of second sample sub-images, and obtaining a third sample position feature of each second sample sub-image in the second sample image, wherein a resolution of each second sample sub-image is equal to the target resolution; for each second sample sub-image, encoding the second sample sub-image by an initial image coding model to obtain a fourth sample image feature of the second sample sub-image, and fusing the fourth sample image feature and the third sample position feature to obtain a fifth sample image feature of the second sample sub-image; decoding the fifth sample image features of the plurality of second sample sub-images by the initial image coding model to obtain a third sample image; based on the third sample image and the second sample image, training the initial image coding model to obtain the trained image coding model.

13. The method of claim 12, wherein, The decoding of the fifth sample image features of the plurality of second sample sub-images by the initial image coding model to obtain a third sample image includes: obtaining a reference sample image, which has a part of the same content as the second sample image; performing feature conversion on the reference sample image by the initial image coding model to obtain a sample query feature; performing feature conversion on the fifth sample image features of the plurality of second sample sub-images by the initial image coding model to obtain a sample key feature and a sample value feature; fusing the sample query feature, the sample key feature and the sample value feature to obtain a sample fusion feature; decoding the sample fusion feature by the initial image coding model to obtain the third sample image.

14. The method of claim 12 or 13, wherein, The third sample position feature of each second sample sub-image in the second sample image is obtained by: for any second sample sub-image, determining coordinate information of the second sample sub-image in the second sample image; based on a preset mapping parameter, mapping the coordinate information to obtain the third sample position feature of the second sample sub-image; The initial image coding model is trained based on the third sample image and the second sample image, to obtain a trained image coding model, including: The initial image coding model and the preset mapping parameter are trained based on the difference between the third sample image and the second sample image.

15. The method of any one of claims 10-14, wherein, The initial processing model is pre-trained, and a pre-training process of the initial processing model includes: Obtain a fourth sample image and obtain a sample image feature of a fifth sample image, the fifth sample image being an image associated with the fourth sample image in content; The fourth sample image is divided to obtain a plurality of fourth sample sub-images, and a fourth sample position feature of each fourth sample sub-image in the fourth sample image is obtained, wherein the resolution of each fourth sample sub-image is equal to the target resolution; For each fourth sample sub-image, the fourth sample sub-image is encoded by the image coding model to obtain a sixth sample image feature of the fourth sample sub-image, and the sixth sample image feature and the fourth sample position feature are fused to obtain a seventh sample image feature of the fourth sample sub-image; The seventh sample image features of the plurality of fourth sample sub-images are processed by the initialized initial processing model to obtain an eighth sample image feature; The initial processing model is pre-trained based on the eighth sample image feature and the sample image feature of the fifth sample image.

16. The method of any one of claims 10-15, wherein, The target task is a multi-modal task, and the data to be processed in the multi-modal task includes the first sample image and first sample non-image data; the second sample image features of the plurality of first sample sub-images are processed by the initial processing model to obtain third sample image features, including: The second sample image features of the plurality of first sample sub-images and first sample data features of the first sample non-image data are processed by the initial processing model to obtain the third sample image features and second sample data features; The initial processing model is trained based on the third sample image features and the target sample image features to obtain a task processing model for processing the target task, including: The initial processing model is trained based on the third sample image features, the target sample image features, the second sample data features, and the first sample data features to obtain the task processing model.

17. A data processing apparatus, comprising: An acquisition module is configured to acquire a first image to be processed in a target task; An image processing module is configured to divide the first image to obtain a plurality of sub-images, and obtain a first position feature of each sub-image in the first image, wherein the resolution of each sub-image is equal to a target resolution of an image that can be processed by a trained image coding model; an encoding module, configured to, for each sub-image, encode the sub-image by using the image coding model to obtain a first image feature of the sub-image, and fuse the first image feature and the first position feature to obtain a second image feature of the sub-image; a feature processing module, configured to process the second image features of the plurality of sub-images by using the trained task processing model to obtain third image features; and a decoding module, configured to decode the third image features by using the image coding model to obtain a second image.

18. A model training apparatus, comprising: an acquisition module, configured to acquire a first sample image; based on a target task, process the first sample image to obtain a target sample image, and acquire a target sample image feature of the target sample image; an image processing module, configured to divide the first sample image to obtain a plurality of first sample sub-images, and acquire a first sample position feature of each first sample sub-image in the first sample image, wherein a resolution of each first sample sub-image is equal to a target resolution of an image that can be processed by a trained image coding model; an encoding module, configured to, for each first sample sub-image, encode the first sample sub-image by using the image coding model to obtain a first sample image feature of the first sample sub-image, and fuse the first sample image feature and the first sample position feature to obtain a second sample image feature of the first sample sub-image; a feature processing module, configured to process the second sample image features of the plurality of first sample sub-images by using an initial processing model to obtain third sample image features; and a first training module, configured to train the initial processing model based on the third sample image features and the target sample image feature to obtain a task processing model for processing the target task.

19. An electronic device, comprising a processor and a memory, wherein the memory stores at least one computer program, the at least one computer program is loaded and executed by the processor to implement the data processing method in any one of claims 1 to 9, or implement the model training method in any one of claims 10 to 16.

20. A computer-readable storage medium, storing at least one computer program, the at least one computer program is loaded and executed by a processor to implement the data processing method in any one of claims 1 to 9, or implement the data processing method in any one of claims 10 to 16.

21. A computer program product, comprising a computer program, the computer program is loaded and executed by a processor to implement the data processing method in any one of claims 1 to 9, or implement the model training method in any one of claims 10 to 16.

Images