Data processing method and device based on artificial intelligence, computer equipment and medium

Abstract

The invention belongs to the technical field of artificial intelligence, and relates to an artificial intelligence-based data processing method, which comprises the following steps of: randomly extracting two image-text pairs from an image-text training data set; creating a blank canvas based on the splicing mode, and performing spatial splicing on the first image and the second image based on the blank canvas to obtain a composite image; processing the first text description and the second text description based on a space template library and a question and answer template library to generate a text space description, and constructing a target data pair based on the composite image and the text space description; based on a data generation mode of the target data pair, performing sample construction on the image-text training data set according to the splicing ratio to obtain a target training data set; and training the language model based on the target training data set to obtain a target language model, and performing business processing based on the target language model. The method can be applied to image processing scenes in the financial science and technology field and the digital medical field, and the accuracy of reply processing of the image data is effectively improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology and can be applied to fields such as fintech and digital healthcare, particularly to data processing methods, devices, computer equipment, and storage media based on artificial intelligence. Background Technology

[0002] In recent years, large-scale multimodal language models (MLLMs) have made significant progress in understanding and fusing visual and textual information, bringing new possibilities to numerous industry applications. However, existing models generally face the serious problem of "spatial hallucination," where the model struggles to accurately determine and express the relative spatial positions of objects in an image when generating descriptions, such as "to the left of XXX" or "in front of XXX." Research indicates that this problem mainly stems from "modal asymmetry" in the training data. Images themselves contain rich, multi-dimensional spatial structural information, but the text descriptions in existing large-scale image-text training datasets are very "sparse" in terms of spatial information. Statistically, only a very small percentage (e.g., 1% - 7%) of text descriptions explicitly contain spatial relation words like "to the left of XXX" or "in front of XXX." This makes it difficult for the model to effectively align the spatial structure in the vision with the sparse descriptions in the language during training, leading to a lack of spatial understanding and the "spatial hallucination" problem of fabricating incorrect spatial positions during reasoning, thus reducing the accuracy of the model's inference.

[0003] In the financial and insurance sector, this lack of spatial understanding can cause numerous inconveniences. For example, in auto insurance claims, customers upload photos of the damaged vehicle scene for claims processing. These photos show a specific spatial relationship between the vehicle and its surroundings, but existing models may fail to accurately describe the relative positions of the damaged parts of the vehicle to surrounding objects (such as trees and buildings). This makes it difficult for claims adjusters to fully and accurately understand the accident scene, affecting the efficiency and accuracy of claims assessments. The problem is equally prominent in the field of digital healthcare. Taking remote analysis as an example, patients upload photos of skin lesions, and doctors rely on model-generated descriptions to assist in medical analysis. If the model cannot accurately express the spatial relationship between the lesion and surrounding normal skin or other landmarks (such as moles or scars), doctors may be unable to accurately analyze the condition, delaying treatment and impacting the patient's health and treatment outcomes.

[0004] Therefore, because existing large-scale multimodal language models lack spatial understanding capabilities, when using large-scale multimodal language models to reason about image data, answers are often fabricated, resulting in low accuracy in image data response processing. Summary of the Invention

[0005] The purpose of this application is to propose a data processing method, apparatus, computer device, and storage medium based on artificial intelligence, in order to solve the technical problem that existing large-scale multimodal language models often fabricate answers when using them to reason about image data due to the lack of spatial understanding capabilities, resulting in low accuracy in image data response processing.

[0006] Firstly, an artificial intelligence-based data processing method is provided, including: Two image-text pairs are randomly selected from a preset image-text training dataset; wherein, the two image-text pairs include a first image-text pair and a second image-text pair, the first image-text pair includes a first image and a first text description, and the second image-text pair includes a second image and a second text description; A blank canvas is created based on a preset splicing mode, and the first image and the second image are spliced ​​together using the blank canvas to obtain a corresponding composite image. Based on a preset spatial template library, the first text description and the second text description are processed to generate a text description corresponding to the composite image; Based on a preset question-and-answer template library, the first text description and the second text description are processed to generate question-and-answer text corresponding to the composite image; Based on the text description and the question-and-answer text, a corresponding text space description is generated, and a corresponding target data pair is constructed based on the composite image and the text space description; Based on the data generation method of the target data pair, the corresponding target training dataset is obtained by constructing samples from the image and text training dataset according to the preset splicing ratio; The target language model is trained based on the target training dataset to obtain a trained target language model, and inference processing is performed on the image data to be processed based on the target language model.

[0007] Secondly, an artificial intelligence-based data processing device is provided, comprising: An extraction module is used to randomly extract two image-text pairs from a preset image-text training dataset; wherein, the two image-text pairs include a first image-text pair and a second image-text pair, the first image-text pair includes a first image and a first text description, and the second image-text pair includes a second image and a second text description; The stitching module is used to create a blank canvas based on a preset stitching mode, and to perform image space stitching processing on the first image and the second image based on the blank canvas to obtain a corresponding composite image. The first generation module is used to process the first text description and the second text description based on a preset spatial template library to generate a text description corresponding to the composite image. The second generation module is used to process the first text description and the second text description based on a preset question and answer template library to generate question and answer text corresponding to the composite image; The first construction module is used to generate a corresponding text space description based on the text description and the question-and-answer text, and to construct a corresponding target data pair based on the composite image and the text space description; The second construction module is used to construct the corresponding target training dataset by performing sample construction on the image and text training dataset according to the data generation method of the target data pair and the preset splicing ratio. The processing module is used to train a preset language model based on the target training dataset to obtain a trained target language model, and to perform inference processing on the image data to be processed based on the target language model.

[0008] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described artificial intelligence-based data processing method.

[0009] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps of the aforementioned artificial intelligence-based data processing method.

[0010] In the aforementioned scheme implemented by the artificial intelligence-based data processing method, apparatus, computer equipment, and storage medium, two image-text pairs are first randomly selected from a preset image-text training dataset. These two image-text pairs include a first image-text pair and a second image-text pair. The first image-text pair includes a first image and a first text description, and the second image-text pair includes a second image and a second text description. Then, a blank canvas is created based on a preset stitching pattern, and the first image and the second image are spatially stitched together using the blank canvas to obtain a corresponding composite image. Afterward, the first text description and the second text description are processed based on a preset spatial template library to generate a text corresponding to the composite image. This description involves processing the first and second text descriptions based on a preset question-and-answer template library to generate question-and-answer text corresponding to the composite image; subsequently, generating a corresponding text space description based on the text descriptions and question-and-answer text, and constructing a corresponding target data pair based on the composite image and the text space description; further, based on the data generation method of the target data pair, constructing a corresponding target training dataset by sampling the image-text training dataset according to a preset splicing ratio; finally, training a preset language model based on the target training dataset to obtain a trained target language model, and performing inference processing on the image data to be processed based on the target language model. Based on the above automated processing flow, this application employs an automated processing method that combines spatial stitching with text spatial description to construct composite image-text data pairs with explicit and deterministic spatial layouts. This processing method requires no manual annotation or advanced generative models, injecting rich structured spatial supervision signals into the training data. It significantly improves the model's spatial reasoning ability during pre-training and fine-tuning stages, while maintaining low cost and high scalability. Furthermore, it effectively suppresses the spatial illusion problem where the target language model trained on the target training dataset fabricates incorrect spatial locations during reasoning. Consequently, by using the trained target language model to perform reasoning processing on the image data to generate responses, the accuracy of response processing for image data can be effectively improved. Attached Figure Description

[0011] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is an exemplary system architecture diagram to which this application can be applied; Figure 2This is a flowchart of an embodiment of the artificial intelligence-based data processing method according to this application; Figure 3 This is a schematic diagram of a structure of an embodiment of the artificial intelligence-based data processing apparatus according to this application; Figure 4 This is a schematic diagram of the structure of one embodiment of the computer device according to this application. Detailed Implementation

[0013] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.

[0014] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0015] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0016] like Figure 1 As shown, system architecture 100 may include terminal device 101, network 102, and server 103. Terminal device 101 may be a laptop 1011, tablet 1012, or mobile phone 1013. Network 102 is used as a medium to provide a communication link between terminal device 101 and server 103. Network 102 may include various connection types, such as wired, wireless communication links, or fiber optic cables, etc.

[0017] Users can use terminal device 101 to interact with server 103 via network 102 to receive or send messages, etc. Various communication client applications can be installed on terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social media platform software, etc.

[0018] Terminal device 101 can be various electronic devices with a display screen and support web browsing. In addition to laptops 1011, tablets 1012, or mobile phones 1013, terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 player (Moving Picture Experts Group Audio Layer IV), a laptop computer, and a desktop computer, etc.

[0019] Server 103 can be a server that provides various services, such as a backend server that provides support for the pages displayed on terminal device 101.

[0020] It should be noted that the artificial intelligence-based data processing method provided in the embodiments of this application is generally executed by a server / terminal device, and correspondingly, the artificial intelligence-based data processing device is generally set in the server / terminal device.

[0021] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0022] Continue to refer to Figure 2 This document illustrates a flowchart of an embodiment of the AI-based data processing method according to this application. The order of steps in the flowchart can be changed, and some steps can be omitted, depending on different needs. The AI-based data processing method provided in this application can be applied to any scenario requiring image processing, and therefore can be applied to products in these scenarios, such as image processing products in the financial insurance field. The AI-based data processing method includes the following steps: Step S201: Randomly select two image-text pairs from the preset image-text training dataset; wherein, the two image-text pairs include a first image-text pair and a second image-text pair, the first image-text pair includes a first image and a first text description, and the second image-text pair includes a second image and a second text description.

[0023] In this embodiment, the data processing method based on artificial intelligence runs on an electronic device (e.g., Figure 1The server / terminal device shown can obtain the image-text training dataset via wired or wireless connection. It should be noted that the aforementioned wireless connection methods may include, but are not limited to, 3G / 4G / 5G connections, WiFi connections, Bluetooth connections, WiMAX connections, Zigbee connections, UWB (ultra-wideband) connections, and other currently known or future-developed wireless connection methods. The executing entity of this application is specifically a data processing system, which can be simply referred to as the system. The aforementioned image-text training dataset is a pre-constructed raw dataset used for training language models. By specifying the storage structure of the image-text training dataset, it is assumed that the image-text training dataset is stored in a certain data structure (such as a list, array, etc.), where each element is an image-text pair, containing an image and its corresponding text description. Then, a random number generator is used to generate two different random numbers within the index range of the image-text training dataset. For example, if the dataset has 1000 image-text pairs with an index range of 0 to 999, two different random numbers are generated, such as 321 and 789. The corresponding image-text pairs are then retrieved from the image-text training dataset based on these two random numbers, and denoted as (…). , ) and( , ), where I represents the image and T represents the corresponding text description.

[0024] Data selection is the starting step in the entire data processing process. By randomly selecting two image-text pairs, the diversity and randomness of the subsequently stitched images and text are ensured, thereby enabling a more comprehensive training of the model's ability to process different image combinations and text descriptions.

[0025] Step S202: Create a blank canvas based on a preset splicing mode, and perform image space splicing processing on the first image and the second image based on the blank canvas to obtain the corresponding composite image.

[0026] In this embodiment, the specific implementation process of creating a blank canvas based on a preset splicing mode and performing image space splicing processing on the first image and the second image based on the blank canvas to obtain the corresponding composite image will be further described in detail in subsequent specific embodiments of this application, and will not be elaborated on here.

[0027] Step S203: Process the first text description and the second text description based on a preset spatial template library to generate a text description corresponding to the composite image.

[0028] In this embodiment, the specific implementation process of processing the first text description and the second text description based on the preset spatial template library to generate a text description corresponding to the composite image will be further described in detail in subsequent specific embodiments of this application, and will not be elaborated on here.

[0029] Step S204: Process the first text description and the second text description based on a preset question-and-answer template library to generate question-and-answer text corresponding to the composite image.

[0030] In this embodiment, the specific implementation process of processing the first text description and the second text description based on the preset question-and-answer template library to generate question-and-answer text corresponding to the composite image will be further described in detail in subsequent specific embodiments of this application, and will not be elaborated on here.

[0031] Step S205: Generate a corresponding text space description based on the text description and the question-and-answer text, and construct a corresponding target data pair based on the composite image and the text space description.

[0032] In this embodiment, after obtaining the composite image, a textual spatial description rich in spatial information that matches the composite image is automatically generated. To achieve this, two parallel operations are set up: a narrative description and a narrative question-and-answer. Parallel operation means that these two processes are carried out simultaneously and independently, each generating corresponding textual content according to its own steps and rules. They mine and textualize the spatial information of the composite image from different perspectives. Furthermore, the independently generated texts related to the spatial information of the composite image (textual description and question-and-answer text) together constitute a description of the composite image. A rich description of spatial information, i.e., textual spatial description. The target data pair constructed based on the aforementioned composite image and text spatial description has the following format: Furthermore, subsequent models are trained using this new dataset, thereby associating the concept of "left" with... By strongly binding the visual features of the left half of the body, its spatial understanding ability is significantly improved.

[0033] Step S206: Based on the data generation method of the target data pair, construct samples from the image and text training dataset according to the preset splicing ratio to obtain the corresponding target training dataset.

[0034] In this embodiment, the specific implementation process of constructing the corresponding target training dataset by generating data based on the target data pair according to the preset splicing ratio of the image and text training dataset will be further described in detail in subsequent specific embodiments of this application, and will not be elaborated on here.

[0035] Step S207: Train the preset language model based on the target training dataset to obtain the trained target language model, and perform inference processing on the image data to be processed based on the target language model.

[0036] In this embodiment, the language model described above can specifically employ large-scale multimodal language models (MLLMs). This can be achieved by applying the constructed target training dataset to the language model's (hereinafter referred to as the model) training process. During training, the model is exposed to both the original data and the newly generated concatenated data. The original data helps the model maintain its general visual language capabilities, learning various common visual features and linguistic descriptions; the concatenated data focuses on enhancing the model's spatial understanding capabilities, enabling it to recognize and process spatial relationships between objects. Furthermore, during training, the number of samples constructed for each concatenation mode N (e.g., horizontal mode only) can be adjusted based on the model's training performance (e.g., performance on the validation set). If the model's spatial understanding capability is insufficient, N is increased appropriately; if the general visual language capability declines, N is decreased. Through continuous adjustment and optimization, the most suitable value of N for the current model and task is found, ensuring the model achieves good performance in both aspects, until a well-trained target language model is obtained.

[0037] Furthermore, the generated target language model can be applied to image processing scenarios in the fintech and digital healthcare fields. Specifically, in the financial insurance sector, the image data to be processed could be photos of vehicle damage submitted by customers in a claims scenario, or real estate images used in the risk assessment phase of financial insurance. In the digital healthcare sector, the image data to be processed could be personal photos taken by patients in remote diagnosis scenarios. Alternatively, the image data to be processed could also be medical images (such as X-rays and CT scans) that need to be processed in medical image analysis in digital healthcare.

[0038] For example, a well-trained target language model can be applied to image-related question-and-answer interaction scenarios in the financial insurance field. A corresponding scenario description includes: In a financial insurance claims scenario, a customer submits photos of a damaged vehicle for car insurance claims. The photos show a black sedan parked on the roadside with obvious collision dents on the front, next to a tree. The customer wants to know the relationship between the damaged location of the vehicle and its surrounding environment, so they ask the newly trained target language model: "In which direction is the damaged front of the vehicle relative to the tree?" After analyzing the image and combining its strong spatial understanding capabilities, the model can quickly determine that the front of the vehicle is to the left of the tree and accurately answer the customer: "The damaged front of the vehicle is to the left of the tree." This helps insurance claims personnel to better understand the accident scene and provides an intuitive basis for subsequent claims assessment. Alternatively, it can also be applied to visual language tasks in the financial insurance field. A corresponding scenario description includes: In the risk assessment stage of financial insurance, a large number of property images need to be evaluated to determine whether there are potential risk factors. The newly trained target language model can undertake image description tasks. For example, inputting a picture of an old residential building into the model, showing obvious cracks in the exterior walls, broken windows, and debris piled up in front of the building, the model can generate a detailed and accurate text description: "This is an old residential building with multiple obvious cracks in the exterior walls, broken windows, and debris piled up in front of the building, which may pose certain safety hazards." Such a description can help risk assessors quickly understand the property's condition, determine its risk level, and improve the efficiency and accuracy of risk assessment.

[0039] Furthermore, the trained target language model can be applied to image-related question-and-answer interaction scenarios in the field of digital healthcare. A corresponding scenario description includes: In a remote analysis scenario in digital healthcare, a patient takes a photo of their skin lesion using their mobile phone and uploads it to a medical platform. The photo shows an irregularly shaped red patch on the patient's arm, surrounded by small papules. The patient wants to understand the specific condition of the lesion and asks the newly trained target language model: "Where is the red patch located in the arm image, and how are the surrounding papules distributed?" The model, utilizing its spatial understanding capabilities, analyzes the image and answers: "The red patch is located in the upper middle part of the arm image, and the surrounding papules are mainly distributed on the right and lower side of the patch, with a relatively dense distribution." This provides important visual information for doctors to conduct remote medical analysis, helping them to analyze the condition more accurately. Alternatively, it can also be applied to visual language task scenarios in the field of digital healthcare. A corresponding scenario description includes: In medical imaging teaching scenarios in digital healthcare, to help medical students understand the structure and morphology of human organs more intuitively and deeply, various medical imaging materials are used, and the model is used for precise interpretation and explanation. Taking brain MRI image teaching as an example, the newly trained model can undertake the tasks of image structure recognition and information generation. When a brain MRI image is input into the newly trained target language model, the model first accurately identifies and locates various brain structures in the image, determining the location of major brain regions such as the cerebral hemispheres, cerebellum, and brainstem, as well as the location of finer structures such as the hippocampus and amygdala. Next, the model further analyzes the morphological characteristics of these structures, such as the symmetry of the cerebral hemispheres, the depth and distribution of sulci and gyri, and the size and shape of the ventricles. Based on these analyses, the model generates detailed explanatory information: "This brain MRI image clearly shows the key structures of the brain. The cerebral hemispheres are symmetrical, with surfaces covered by sulci of varying depths and convex gyri, a unique morphology that increases the area of ​​the cerebral cortex. The hippocampus, located deep within the cerebral hemispheres, is shaped like a seahorse and plays a crucial role in memory formation and consolidation. The brainstem, as an important passage connecting the brain and spinal cord, consists of the midbrain, pons, and medulla oblongata, and its relatively elongated shape can be seen in the image. The cerebellum, located at the lower back of the brain, has many parallel cerebellar sulci and is primarily responsible for coordinating body movement and maintaining balance." Such visual-language task processing results can provide medical students with vivid and accurate image interpretation, helping them quickly grasp knowledge of brain structure and improving the effectiveness and efficiency of medical imaging learning.

[0040] This application first randomly selects two image-text pairs from a preset image-text training dataset; wherein the two image-text pairs include a first image-text pair and a second image-text pair, the first image-text pair including a first image and a first text description, and the second image-text pair including a second image and a second text description; then, a blank canvas is created based on a preset stitching mode, and the first image and the second image are subjected to image spatial stitching processing based on the blank canvas to obtain a corresponding composite image; then, the first text description and the second text description are processed based on a preset spatial template library to generate a text description corresponding to the composite image; and finally, a question-and-answer template library is used to process the first text description and the second text description. The first text description and the second text description are processed to generate question-and-answer text corresponding to the composite image; subsequently, a corresponding text space description is generated based on the text description and the question-and-answer text, and a corresponding target data pair is constructed based on the composite image and the text space description; further, based on the data generation method of the target data pair, the image-text training dataset is sampled according to a preset splicing ratio to obtain a corresponding target training dataset; finally, a preset language model is trained based on the target training dataset to obtain a trained target language model, and inference processing is performed on the image data to be processed based on the target language model. Based on the above automated processing flow, this application employs an automated processing method that combines spatial stitching with text spatial description to construct composite image-text data pairs with explicit and deterministic spatial layouts. This processing method requires no manual annotation or advanced generative models, injecting rich structured spatial supervision signals into the training data. It significantly improves the model's spatial reasoning ability during pre-training and fine-tuning stages, while maintaining low cost and high scalability. Furthermore, it effectively suppresses the spatial illusion problem where the target language model trained on the target training dataset fabricates incorrect spatial locations during reasoning. Consequently, by using the trained target language model to perform reasoning processing on the image data to generate responses, the accuracy of response processing for image data can be effectively improved.

[0041] In some alternative implementations, step S202 includes the following steps: Randomly select a splicing pattern from the preset splicing pattern set.

[0042] In this embodiment, a predefined set of splicing patterns is defined, which includes at least two splicing patterns: "horizontal splicing" and "vertical splicing". Specifically, a random number can be generated using a random number generator, and a splicing pattern can be randomly selected from the splicing pattern set based on this random number. For example, if the random number is between 0 and 0.5, "horizontal splicing" is selected, and if it is between 0.5 and 1, "vertical splicing" is selected.

[0043] Obtain the first size corresponding to the first image and the second size corresponding to the second image.

[0044] In this embodiment, for the first image Use image processing tools (such as image reading functions) to obtain its first size, including the first width. and the first height For example, by reading the pixel information of the first image, the width is obtained by counting the number of pixels in the horizontal direction, and the height is obtained by counting the number of pixels in the vertical direction. Similarly, for the second image... Get its width and height Obtaining the dimensions of the two images is necessary for creating a canvas of the appropriate size later, ensuring that the two images can be completely stitched together on the canvas, while avoiding image distortion or exceeding the canvas area.

[0045] A corresponding blank canvas is created based on the first size, the second size, and the splicing pattern.

[0046] In this embodiment, the specific implementation process of creating a corresponding blank canvas based on the first size, the second size and the splicing mode will be further described in detail in subsequent specific embodiments of this application, and will not be elaborated on here.

[0047] Based on the splicing mode, the first image is pasted to a designated position on the blank canvas, and based on the designated position, the second image is pasted to the corresponding position on the blank canvas to obtain the processed image data.

[0048] In this embodiment, the image pasting process includes: if "horizontal stitching" is selected: pasting the first image... Paste onto blank canvas The (0, 0) position is the top left corner of the blank canvas. You can use the image pasting function from an image processing library to paste the image. Copy pixel data to The corresponding position. And the second image. Paste to of( , 0) position, i.e. Position the image on the right to ensure that the two images are closely stitched together horizontally.

[0049] If "vertical stitching" is selected: the first image Paste to The (0, 0) position is the top left corner of the canvas. The second image... Paste onto blank canvas of (0, Location, i.e. At the bottom, ensure that the two images are closely stitched together vertically.

[0050] Image pasting involves precisely placing two original images onto a pre-created canvas according to a selected stitching pattern and calculated positions, forming a composite image with a defined spatial relationship. This step is the core operation of image stitching, directly affecting the quality of the composite image and the accuracy of its spatial layout.

[0051] The image data is used as the composite image.

[0052] In this embodiment, after the above image pasting operation, the resulting image is... This is called a "composite image". .at this time, From China All visual content is spatially located from The specific orientation of all visual content (such as left or top) depends on the selected splicing pattern.

[0053] This application randomly selects a splicing pattern from a preset set of splicing patterns; then obtains the first size corresponding to the first image and the second size corresponding to the second image; subsequently, it creates a corresponding blank canvas based on the first size, the second size, and the splicing pattern; next, it pastes the first image to a designated position on the blank canvas based on the splicing pattern, and pastes the second image to a corresponding position on the blank canvas based on the designated position, obtaining processed image data; finally, it uses the image data as a composite image. Based on the above processing flow, this application constructs a blank canvas according to the selected splicing pattern and the sizes of the two images, and then accurately places the two original images on the created blank canvas according to the selected splicing pattern and the calculated position, forming a composite image with a defined spatial relationship, ensuring the intelligence of the generated composite image. Furthermore, the generated composite image integrates the visual information of the two original images, forming an image with a new spatial layout, which is beneficial for subsequent text spatial description stages and provides accurate training samples with spatial information for the model.

[0054] In some optional implementations of this embodiment, creating a corresponding blank canvas based on the first size, the second size, and the splicing mode includes the following steps: The first width and first height of the first image are obtained based on the first size, and the second width and second height of the second image are obtained based on the second size.

[0055] In this embodiment, the first width and first height of the first image can be extracted by analyzing the first dimension, and the second width and second height of the second image can be extracted by analyzing the second dimension.

[0056] Obtain the data calculation strategy that matches the splicing pattern.

[0057] In this embodiment, the above data calculation strategy includes: 1) If the selected splicing mode is "horizontal splicing": calculate the width of the new canvas. That is, the sum of the widths of the two images. Calculate the height of the new canvas. This means taking the maximum height of the two images to ensure that both images are fully displayed on the canvas. Create a canvas with a size of ( , blank canvas You can use the function to create a blank image in the image processing library to set the background color of the canvas (such as white).

[0058] 2) If the selected stitching mode is "Vertical Stitching": Calculate the width of the new canvas. This means taking the maximum width of the two images. Calculate the height of the new canvas. That is, the sum of the heights of the two images. Create a shape with a size of ( , blank canvas .

[0059] The canvas width is calculated by using the data calculation strategy to calculate the first width and the second width, and the canvas height is calculated by using the first height and the second height.

[0060] In this embodiment, the canvas width can be calculated by performing canvas width calculation on the first width and the second width based on the strategy content of the above data calculation strategy, and the canvas height can be calculated by performing canvas height calculation on the first height and the second height.

[0061] Based on a preset image processing library, the canvas width and canvas height are processed to create the canvas, resulting in the corresponding canvas data.

[0062] In this embodiment, it can be achieved by creating a size of ( , blank canvas Specifically, you can use the "Create Blank Image" function in the image processing library to set the background color of the canvas (such as white).

[0063] Use the canvas data as the blank canvas.

[0064] This application obtains the first width and first height of a first image based on a first size, and the second width and second height of a second image based on a second size; then it obtains a data calculation strategy matching the splicing mode; and calculates the canvas width and height based on the first and second widths using the data calculation strategy; subsequently, it performs canvas creation processing on the canvas width and height using a preset image processing library to obtain the corresponding canvas data; finally, it uses the canvas data as a blank canvas. Based on the above processing flow, this application provides a suitable space for image splicing according to the selected splicing mode and the sizes of the two images. Furthermore, by reasonably calculating the canvas size, it can effectively ensure that the spliced ​​image can be displayed completely and clearly, and has a correct spatial layout.

[0065] In some alternative implementations, step S203 includes the following steps: Call the preset space template library.

[0066] In this embodiment, a "spatial template library" is created based on actual business needs. This library can be a list or a collection. It can be categorized and stored according to different splicing modes by collecting a large number of sentence templates describing spatial relationships. For example, for the "horizontal splicing" mode, it can include "left-side display". The right side shows "." On the left, On the right. Templates such as "etc."; for "vertical splicing" mode, it can include "show above". The following is displayed "." Above, Below. (e.g., templates)

[0067] The template library is maintained to provide a rich variety of sentence templates describing spatial relationships, which will be used to generate text descriptions that match composite images. By categorizing and storing templates with different splicing patterns, the accuracy and efficiency of template selection can be improved.

[0068] Obtain the spatial template corresponding to the splicing mode from the spatial template library.

[0069] In this embodiment, a spatial template can be randomly selected from the spatial template library based on the selected splicing mode. For example, if the "horizontal splicing" mode is selected, a template is randomly selected from the horizontal splicing template library.

[0070] The first text description and the second text description are filled into the corresponding positions in the space template to obtain the filled space template.

[0071] In this embodiment, the original text (First text description) and (Second text description) Fill in the placeholders in the selected space template respectively. and For example, if the selected template is "Show on the left". The right side shows .",Will Fill in " , Fill in Generate T_stitch (description), such as "left side display". The right side shows "Furthermore, the filled space template is used as the corresponding text description."

[0072] The filled space template is used as the text description.

[0073] This application calls a preset spatial template library; then retrieves a spatial template corresponding to the splicing mode from the library; subsequently, it fills the corresponding positions within the spatial template with the first and second text descriptions to obtain the filled spatial template; and finally, it uses the filled spatial template as the text description. Based on the above processing flow, the template selection and filling in this application are based on the splicing mode of the composite image, selecting a suitable spatial template from the spatial template library, and filling the spatial template with the original text information to generate a structured text description rich in spatial information. Furthermore, the generated text description accurately reflects the spatial layout of the two original images in the composite image, thus providing valuable training data for the model.

[0074] In some alternative implementations, step S204 includes the following steps: The first text description is subjected to noun extraction to obtain a corresponding first noun list, and the second text description is subjected to noun extraction to obtain a corresponding second noun list.

[0075] In this embodiment, the original first text description can be used. Natural language processing is performed, using part-of-speech tagging tools (such as word segmentation and part-of-speech tagging libraries) to segment the text into words and label the part of speech of each word. Words with the part of speech of nouns are then selected to form a list of concrete, visible object nouns (the first noun list). For example, if The phrase "a cat on a table" could be used to extract a list of nouns, such as {"cat", "table"}. Similarly, the original second text description... Perform the same process to obtain the second list of nouns. .

[0076] Noun extraction involves retrieving information about specific objects from the original text, which will have specific spatial locations in the composite image. Extracting nouns provides the foundation for generating subsequent question-and-answer pairs, enabling these pairs to perform spatial reasoning based on specific objects in the composite image.

[0077] A first noun is randomly selected from the first list of nouns, and a second noun is randomly selected from the second list of nouns.

[0078] In this embodiment, a random number generator can be used to generate a number from the aforementioned first list of terms. Randomly select a noun Similarly, from the second list of terms above... Randomly select a noun The question-answer pair generation process involves randomly selecting nouns from two different texts to prepare for generating subsequent question-answer pairs with randomness and diversity. This random sampling method increases the coverage of question-answer pairs, enabling the model to learn spatial relationships between different objects.

[0079] Obtain the specified question and answer template from the question and answer template library.

[0080] In this embodiment, a "question-answer template library" is pre-created, containing various question-answer templates for spatial reasoning, such as "From the observer's perspective, Is it located in "On the left side?" exist "Is it above?" etc. Then, a template is randomly selected from the question and answer template library as the specified question and answer template.

[0081] Based on the spatial layout of the composite image and the specified question-and-answer template, question-and-answer generation processing is performed on the first noun and the second noun to obtain corresponding question-and-answer pairs.

[0082] In this embodiment, the answer is automatically inferred based on the spatial layout of the composite image (determined by the stitching mode). For example, if the "horizontal stitching" mode is selected, and The corresponding object is on the left. The corresponding object is on the right. Regarding the template "from the observer's perspective," Is it located in "On the left side?", the answer is "yes". And generate... (Question and answer correspondence), such as (Q: "From the observer's perspective, is the cat located to the left of the dog?", A: "Yes").

[0083] The question-and-answer pair is used as the question-and-answer text.

[0084] This application obtains a first noun list by extracting nouns from a first text description and a second noun list by extracting nouns from a second text description. Then, it randomly selects a first noun from the first noun list and a second noun from the second noun list. Next, it retrieves a specified question-and-answer template from a question-and-answer template library. Subsequently, based on the spatial layout of the composite image and the specified question-and-answer template, it performs question-and-answer generation processing on the first and second nouns to obtain corresponding question-and-answer pairs. Finally, it uses the question-and-answer pairs as the question-and-answer text. Based on the above processing flow, this application generates question-and-answer pairs for spatial reasoning by randomly extracting nouns from two different texts, selecting appropriate templates from the question-and-answer template library based on the extracted nouns and the spatial layout of the composite image, and automatically inferring the answers. These question-and-answer pairs can serve as high-quality data for supervised fine-tuning, thereby helping the model improve its spatial reasoning ability.

[0085] In some optional implementations of this embodiment, step S206 includes the following steps: Obtain the training target corresponding to the language model, and determine the splicing ratio corresponding to the language model based on the training target.

[0086] In this embodiment, when determining the splicing ratio λ (reflecting the spatial supervision strength) based on the training objectives of the language model, the following factors can be considered comprehensively, taking into account the specific task scenario, data characteristics, and model performance requirements: 1. Define the core spatial understanding requirements of the task. For high-precision spatial positioning tasks: If the task requires the model to accurately identify the position of objects (such as vehicle and pedestrian positioning in autonomous driving, and part position determination in industrial inspection), spatial supervision needs to be strengthened. In this case, a larger λ should be set, for example, λ≥1 (or 1), so that the stitched data dominates during training, forcing the model to learn the spatial relationships between objects in depth and improving positioning accuracy. For general visual understanding tasks: If the task focuses more on the extraction of general visual features (such as image classification and image retrieval) and has a weaker dependence on spatial relationships, a smaller λ should be set, for example, λ∈(0,0.5), to avoid excessive interference from spatial supervision on the model's learning of general visual features and to maintain the model's generalization ability. For multi-task balancing scenarios: If the task needs to handle both spatially related and general visual tasks simultaneously (such as identifying both personnel position and personnel movement in intelligent security), λ can be adjusted experimentally to find a balance point. For example, start with λ=0.5, gradually adjust and observe the model's performance on both types of tasks, and finally determine the optimal value. Based on the splicing ratio and the image and text training dataset, a specified number of spliced ​​samples to be generated is determined.

[0087] 2. Analyzing Data Characteristics and Spatial Information Richness: Data with abundant spatial information: If the original dataset contains a large number of clear spatial relationships (e.g., objects arranged in an orderly manner in a scene image), λ can be appropriately reduced, as the original data already provides sufficient spatial supervision. For example, λ∈(0.2,0.5) avoids introducing redundant information through data stitching. Data with scarce spatial information: If the spatial relationships in the original data are ambiguous or missing (e.g., randomly photographed object images), spatial information needs to be supplemented by data stitching. In this case, λ should be increased, for example, λ≥0.8, to strengthen the model's learning of spatial relationships.

[0088] 3. Integrate model performance requirements with experimental performance metrics: Adjust λ based on the core metrics relevant to the task (e.g., mean accuracy (mAP) for localization and accuracy for classification). For example, if the mAP for localization is low, gradually increase λ and observe the changes to find the range with the most significant improvement. Cross-validation and tuning: Divide the dataset into training, validation, and test sets, and adjust λ through cross-validation during training. For example, test the model's performance on the validation set with different λ values, and select the λ value that optimizes the validation set performance as the final value. Gradual adjustment strategy: Start with a conservative value (e.g., λ=0.3), gradually increase or decrease λ, and observe the changes in the model's loss function value, accuracy, and other metrics on the training and validation sets. If increasing λ significantly improves the model's performance on spatially relevant tasks without significantly impairing general vision task performance, it can be increased further; otherwise, it should be decreased.

[0089] 4. Consider Model Complexity and Computational Resources: Model complexity: If the model structure is simple (e.g., lightweight CNN), an excessively large λ may lead to overfitting of the spatial data; in this case, a smaller λ should be chosen. If the model complexity is high (e.g., deep Transformer), a larger λ can be tried to fully utilize the model's capacity to learn spatial information. Computational resource constraints: Data generation and training consume additional computational resources. If resources are limited, a trade-off must be made between λ and training efficiency. For example, a smaller λ can be chosen to quickly train the base model, and then λ can be gradually increased for fine-tuning as needed.

[0090] Based on the splicing ratio and the image and text training dataset, a specified number of spliced ​​samples to be generated is determined.

[0091] In this embodiment, the specific implementation process of determining the specified number of spliced ​​samples to be generated based on the splicing ratio and the image and text training dataset will be further described in detail in subsequent specific embodiments of this application, and will not be elaborated on here.

[0092] Based on the data generation method of the target data pairs, the image-text training dataset is processed to generate samples, resulting in target spliced ​​samples corresponding to the specified number. In this embodiment, after determining the specified number N of spliced ​​samples to be generated, spliced ​​samples are generated according to the same generation method as the target data pairs obtained after processing. For each splicing mode (such as horizontal and vertical), N spliced ​​samples are constructed respectively. Specifically, two samples (image-text pairs) are randomly selected from the original image-text training dataset and spliced ​​according to the set splicing mode. For example, the horizontal splicing mode splices the two samples in the horizontal direction, and the vertical splicing mode splices them in the vertical direction. This generates a total of (N horizontal and N vertical) new target splicing samples.

[0093] Obtain the remaining original samples after processing the image and text training dataset.

[0094] In this embodiment, since each spliced ​​sample consumes 2 original samples, and assuming two splicing modes are used, a total of 4N original samples are consumed in the above image and text training dataset. Therefore, the number of original samples remaining after sample generation processing in this original image and text training dataset is calculated as follows: , This represents the total number of samples in the image and text training dataset.

[0095] The target spliced ​​sample is combined with the original sample to construct the corresponding target training dataset.

[0096] In this embodiment, after generating new target spliced ​​samples, they are combined with the remaining original samples to form the target training dataset. The number of remaining original samples... (Because two splicing modes are used, a total of 4N original samples are consumed), the total number of samples in the target splicing sample of the new splicing. This combines the old and new data according to the splicing ratio λ, preparing them for subsequent model training.

[0097] This application obtains the training objective corresponding to the language model and determines the splicing ratio corresponding to the language model based on the training objective. Then, based on the splicing ratio and the image-text training dataset, a specified number of spliced ​​samples to be generated is determined. Next, the image-text training dataset is processed to generate samples according to the data generation method of the target data pair, resulting in target spliced ​​samples corresponding to the specified number. Subsequently, the remaining original samples after the sample generation processing of the image-text training dataset are obtained. Finally, the target spliced ​​samples and the original samples are combined to construct the corresponding target training dataset. Based on the above processing flow, this application determines the splicing ratio based on the training objective of the language model, determines the specified number of spliced ​​samples to be generated based on the splicing ratio and the image-text training dataset, then processes the image-text training dataset to generate target spliced ​​samples according to the data generation method of the target data pair, and finally combines the remaining original samples after the sample generation processing of the image-text training dataset with the obtained target spliced ​​samples to achieve rapid and intelligent construction of the required target training dataset. By defining the splicing ratio to balance the proportion of newly generated target spliced ​​samples and original data in the training batch, the optimal balance can be found between improving spatial understanding and maintaining general visual language ability. This allows the model to learn both types of data simultaneously according to different training needs and data characteristics, thus maintaining general visual language ability and improving spatial understanding, thereby effectively improving the model's performance and generalization ability.

[0098] In some optional implementations of this embodiment, determining the specified number of spliced ​​samples to be generated based on the splicing ratio and the image-text training dataset includes the following steps: Obtain the total number of samples in the image and text training dataset.

[0099] In this embodiment, the total number of samples refers to the total size T of the above-mentioned image and text training dataset obtained through statistics.

[0100] Call the preset splicing ratio calculation formula.

[0101] In this embodiment, the splicing ratio Defined as the ratio of new to old data volume, the above splicing ratio calculation formula specifically includes: The parameters are explained as follows: : This refers to the splicing ratio, which is an adjustable hyperparameter used to control the intensity of spatial supervision. : Indicates the total number of newly generated structured spliced ​​samples. : Represents the number of original samples remaining in the original dataset that were not used for concatenation. T: Represents the total number of samples in the original dataset. N: Represents the number of samples constructed for each concatenation pattern (e.g., horizontal only).

[0102] Based on the splicing ratio calculation formula, algebraic operations are performed on the total number of samples and the splicing ratio to obtain the corresponding solution.

[0103] In this embodiment, after determining the splicing ratio and the total number of samples in the original image-text training dataset, the value of N can be solved using algebraic operations. Specifically, given λ and T, according to the formula 2N = λ(T) 4N), expanding it gives 2N = λ(T) 4λN), which can be further rearranged into 2N+4λN=λT, that is, N(2+4λ)=λT, and thus N=λT / (2+4λ). Therefore, determining λ first makes it easier to calculate N, that is, the solution result, and serve as the specified number of spliced ​​samples to be generated.

[0104] The solution result is used as the specified number of spliced ​​samples to be generated.

[0105] This application obtains the total number of samples in the image and text training dataset; then calls a preset splicing ratio calculation formula; subsequently, it performs algebraic operations on the total number of samples and the splicing ratio based on the splicing ratio calculation formula to obtain the corresponding solution result; and finally, the solution result is used as the specified number of spliced ​​samples to be generated. Based on the above processing flow, this application obtains the total number of samples in the image and text training dataset, then performs algebraic operations on the total number of samples and the splicing ratio based on the splicing ratio calculation formula, and uses the obtained solution result as the specified number of spliced ​​samples to be generated, thereby achieving efficient and accurate determination of the specified number of spliced ​​samples to be generated, ensuring the accuracy and intelligence of the generated specified number.

[0106] In some alternative implementations, the user information obtained is subject to user consent and complies with relevant laws and policies.

[0107] Furthermore, any software tools or components not belonging to our company that appear in the embodiments of this application are merely illustrative examples and do not represent actual use.

[0108] Furthermore, the core objective of this application is to provide an innovative, annotation-free, and plug-and-play multimodal data augmentation method. This method addresses the problems of sparse spatial information in training data and severe spatial illusion in models mentioned in the background art by automatically constructing data pairs with consistent text-image spatial structure. Moreover, this application does not rely on expensive manual annotation or advanced generative models, but rather cleverly constructs spatial supervision signals within the data processing flow. Its core innovations are as follows: Image-Spatial Stitching: This invention first selects two or more image-text pairs from a dataset and stitches their images along a spatial axis (e.g., a horizontal or vertical axis). This creates a completely new "stitched image" with an explicit, program-controlled, deterministic spatial layout (e.g., image A must be to the left of image B).

[0109] Text-Spatial Telling: After obtaining a composite image with a well-defined spatial layout, this invention automatically generates a corresponding text description rich in spatial information based on this known layout. This process requires no model reasoning or manual writing.

[0110] Dual text generation mechanism: The "text spatial description" step of this invention incorporates two complementary implementations to provide multi-dimensional spatial supervision: (a) Structured Description (Tell the Caption): This invention uses a "spatial template library" to combine the original text descriptions of two images ( and Fill the template with the text. For example, generate "Left side display: { }, while the right side shows: { Such descriptions explicitly associate spatial terms ("left side", "right side") with corresponding visual regions.

[0111] (b) Spatial Question and Answer Pairs: This invention further extracts information from the original text. and Extract key object names ( and It also automatically generates spatial reasoning question-and-answer pairs. For example, it generates the question: "From the observer's perspective, { Is it located in { "The left side of}?" Since the spatial layout is constructed by this invention itself, the answer ("yes" or "no") is "self-consistent and definite".

[0112] Label-free and highly scalable: The entire process (image stitching, text template filling, question-answer pair generation) is fully automated, without any manual costs or complex model dependencies. This makes it highly scalable and allows for seamless integration into any existing multimodal model pre-training or supervised fine-tuning (SFT) workflow.

[0113] Furthermore, this application has the following significant beneficial effects: Significantly alleviates spatial illusion: By providing the model with a massive amount of training samples with explicit spatial structures ("left-right", "top-bottom") and corresponding spatial descriptions ("left side is...", "top is..."), this invention forces the model to learn the precise alignment relationship between visual layout and spatial vocabulary, thereby effectively suppressing the "spatial illusion" problem in which the model fabricates incorrect spatial locations during inference.

[0114] Completely free of manual annotation: This invention utilizes a "programmatic construction" approach, automatically creating a spatial layout through image stitching, and then automatically generating text descriptions and question-and-answer pairs based on this known layout. The entire process is self-consistent and automated, completely eliminating reliance on expensive and time-consuming manual crowdsourced annotation.

[0115] High scalability and easy integration: As a "plug-and-play" data preprocessing module, this invention does not modify any model architecture and does not depend on any advanced generative model. It can be seamlessly integrated into the data pipeline of pre-training and supervised fine-tuning (SFT) of any multimodal model and is suitable for image and text datasets of any size.

[0116] Improving Model Robustness and Efficiency: By stitching multiple images into a single image, this invention increases the information density of a single sample, forcing the model to understand more complex visual scenes, thereby improving the model's robustness. Simultaneously, during training, the model processes the content of multiple images in a single step, indirectly improving data throughput efficiency.

[0117] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0118] It should be emphasized that, to further ensure the privacy and security of the aforementioned target training dataset, the target training dataset can also be stored in a node of a blockchain.

[0119] The blockchain referred to in this application is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.

[0120] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0121] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware through computer-readable instructions. These computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above methods. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, optical disk, or read-only memory (ROM), or random access memory (RAM).

[0122] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0123] Further reference Figure 3 As a response to the above Figure 2 To implement the method shown, this application provides an embodiment of a data processing device based on artificial intelligence, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0124] like Figure 3As shown, the artificial intelligence-based data processing device 300 described in this embodiment includes: an extraction module 301, a splicing module 302, a first generation module 303, a second generation module 304, a first construction module 305, a second construction module 306, and a processing module 307. Wherein: Extraction module 301 is used to randomly extract two image-text pairs from a preset image-text training dataset; wherein, the two image-text pairs include a first image-text pair and a second image-text pair, the first image-text pair includes a first image and a first text description, and the second image-text pair includes a second image and a second text description; The splicing module 302 is used to create a blank canvas based on a preset splicing mode, and to perform image space splicing processing on the first image and the second image based on the blank canvas to obtain a corresponding composite image. The first generation module 303 is used to process the first text description and the second text description based on a preset spatial template library to generate a text description corresponding to the composite image. The second generation module 304 is used to process the first text description and the second text description based on a preset question and answer template library to generate question and answer text corresponding to the composite image. The first construction module 305 is used to generate a corresponding text space description based on the text description and the question-and-answer text, and to construct a corresponding target data pair based on the composite image and the text space description; The second construction module 306 is used to construct a corresponding target training dataset by performing sample construction on the image and text training dataset according to a preset splicing ratio based on the data generation method of the target data pair. The processing module 307 is used to train a preset language model based on the target training dataset to obtain a trained target language model, and to perform inference processing on the image data to be processed based on the target language model.

[0125] In some optional implementations of this embodiment, the splicing module 302 includes: The selection submodule is used to randomly select a splicing pattern from a preset set of splicing patterns; The first acquisition submodule is used to acquire a first size corresponding to the first image and a second size corresponding to the second image; Create a submodule to create a corresponding blank canvas based on the first size, the second size, and the splicing pattern; The processing submodule is used to paste the first image to a specified position on the blank canvas based on the splicing mode, and paste the second image to the corresponding position on the blank canvas based on the specified position, so as to obtain the processed image data; The first determining submodule is used to treat the image data as the composite image.

[0126] In some optional implementations of this embodiment, creating a submodule for creating a corresponding blank canvas based on the first size, the second size, and the splicing mode includes: The first acquisition unit is configured to acquire the first width and first height of the first image based on the first size, and to acquire the second width and second height of the second image based on the second size; The second acquisition unit is used to acquire a data calculation strategy that matches the splicing pattern; The calculation unit is used to calculate the canvas width by calculating the first width and the second width based on the data calculation strategy, and to calculate the canvas height by calculating the first height and the second height. A creation unit is used to perform canvas creation processing on the canvas width and the canvas height based on a preset image processing library to obtain the corresponding canvas data; The first determining unit is used to use the canvas data as the blank canvas.

[0127] In some optional implementations of this embodiment, the first generation module 303 includes: Calling a submodule is used to call a preset space template library; The second acquisition submodule is used to acquire a spatial template corresponding to the splicing mode from the spatial template library; The fill submodule is used to fill the first text description and the second text description into the corresponding positions in the space template to obtain the filled space template; The second determining submodule is used to use the filled space template as the text description.

[0128] In some optional implementations of this embodiment, the second generation module 304 includes: The extraction submodule is used to extract nouns from the first text description to obtain a corresponding first noun list, and to extract nouns from the second text description to obtain a corresponding second noun list; The extraction submodule is used to randomly extract a first noun from the first noun list and randomly extract a second noun from the second noun list; The third acquisition submodule is used to acquire a specified question and answer template from the question and answer template library; The first generation submodule is used to perform question-and-answer generation processing on the first noun and the second noun based on the spatial layout of the composite image and the specified question-and-answer template, so as to obtain the corresponding question-and-answer pairs. The third determining submodule is used to treat the question-answer pair as the question-answer text.

[0129] In some optional implementations of this embodiment, the second construction module 306 includes: The fourth determination submodule is used to obtain the training target corresponding to the language model and determine the splicing ratio corresponding to the language model based on the training target; The fifth determining submodule is used to determine the specified number of spliced ​​samples to be generated based on the splicing ratio and the image and text training dataset; The second generation submodule is used to perform sample generation processing on the image and text training dataset based on the data generation method of the target data pair, so as to obtain the target splicing sample corresponding to the specified number. The fourth acquisition submodule is used to acquire the remaining original samples after the image and text training dataset has been processed to generate samples; A submodule is constructed to combine the target spliced ​​sample with the original sample to construct the corresponding target training dataset.

[0130] In some optional implementations of this embodiment, the fifth determining submodule includes: The third acquisition unit is used to acquire the total number of samples in the image and text training dataset; The calling unit is used to call the preset splicing ratio calculation formula; The solution unit is used to perform algebraic operations on the total number of samples and the splicing ratio based on the splicing ratio calculation formula to obtain the corresponding solution result; A determining unit is used to use the solution result as a specified number of spliced ​​samples to be generated.

[0131] To address the aforementioned technical problems, embodiments of this application also provide a computer device. Please refer to [link / reference needed] for details. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0132] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected via a system bus. It should be noted that only the computer device 4 with components 41-43 is shown in the figure; however, it should be understood that it is not required to implement all the shown components, and more or fewer components can be implemented alternatively. Those skilled in the art will understand that the computer device described here is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0133] The computer device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device can interact with the user via a keyboard, mouse, remote control, touchpad, or voice control.

[0134] The memory 41 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 41 may be an internal storage unit of the computer device 4, such as the hard disk or memory of the computer device 4. In other embodiments, the memory 41 may also be an external storage device of the computer device 4, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 4. Of course, the memory 41 may also include both the internal storage unit and its external storage device of the computer device 4. In this embodiment, the memory 41 is typically used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions for data processing methods based on artificial intelligence. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or will be output.

[0135] In some embodiments, the processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor 42 is typically used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to execute computer-readable instructions stored in the memory 41 or to process data, for example, to execute computer-readable instructions of the artificial intelligence-based data processing method.

[0136] The network interface 43 may include a wireless network interface or a wired network interface, which is typically used to establish communication connections between the computer device 4 and other electronic devices.

[0137] This application also provides another embodiment, namely, providing a computer-readable storage medium storing computer-readable instructions that can be executed by at least one processor to cause the at least one processor to perform the steps of the artificial intelligence-based data processing method described above.

[0138] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0139] Obviously, the embodiments described above are only some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms; rather, the purpose of providing these embodiments is to provide a more thorough and comprehensive understanding of the disclosure of this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of patent protection of this application.

Claims

1. A data processing method based on artificial intelligence, characterized in that, Includes the following steps: Two image-text pairs are randomly selected from a preset image-text training dataset; wherein, the two image-text pairs include a first image-text pair and a second image-text pair, the first image-text pair includes a first image and a first text description, and the second image-text pair includes a second image and a second text description; A blank canvas is created based on a preset splicing mode, and the first image and the second image are spliced ​​together using the blank canvas to obtain a corresponding composite image. Based on a preset spatial template library, the first text description and the second text description are processed to generate a text description corresponding to the composite image; Based on a preset question-and-answer template library, the first text description and the second text description are processed to generate question-and-answer text corresponding to the composite image; Based on the text description and the question-and-answer text, a corresponding text space description is generated, and a corresponding target data pair is constructed based on the composite image and the text space description; Based on the data generation method of the target data pair, the corresponding target training dataset is obtained by constructing samples from the image and text training dataset according to the preset splicing ratio; The target language model is trained based on the target training dataset to obtain a trained target language model, and inference processing is performed on the image data to be processed based on the target language model.

2. The data processing method based on artificial intelligence according to claim 1, characterized in that, The step of creating a blank canvas based on a preset splicing pattern, and then performing image space splicing processing on the first image and the second image based on the blank canvas to obtain a corresponding composite image, specifically includes: Randomly select a splicing pattern from the preset splicing pattern set; Obtain the first size corresponding to the first image and the second size corresponding to the second image; Create a corresponding blank canvas based on the first size, the second size, and the splicing pattern; Based on the splicing mode, the first image is pasted to a designated position on the blank canvas, and based on the designated position, the second image is pasted to the corresponding position on the blank canvas to obtain the processed image data; The image data is used as the composite image.

3. The data processing method based on artificial intelligence according to claim 2, characterized in that, The step of creating a corresponding blank canvas based on the first size, the second size, and the splicing pattern specifically includes: The first width and first height of the first image are obtained based on the first size, and the second width and second height of the second image are obtained based on the second size; Obtain the data calculation strategy that matches the splicing pattern; The canvas width is calculated by the first width and the second width based on the data calculation strategy, and the canvas height is calculated by the first height and the second height. Based on a preset image processing library, the canvas width and canvas height are processed to create the canvas data. Use the canvas data as the blank canvas.

4. The data processing method based on artificial intelligence according to claim 1, characterized in that, The step of processing the first text description and the second text description based on a preset spatial template library to generate a text description corresponding to the composite image specifically includes: Call the preset space template library; Obtain the spatial template corresponding to the splicing mode from the spatial template library; Fill the corresponding positions in the spatial template with the first text description and the second text description to obtain the filled spatial template; The filled space template is used as the text description.

5. The data processing method based on artificial intelligence according to claim 1, characterized in that, The step of processing the first text description and the second text description based on a preset question-and-answer template library to generate question-and-answer text corresponding to the composite image specifically includes: The first text description is subjected to noun extraction to obtain a corresponding first noun list, and the second text description is subjected to noun extraction to obtain a corresponding second noun list; Randomly select a first noun from the first noun list, and randomly select a second noun from the second noun list; Obtain the specified question and answer template from the question and answer template library; Based on the spatial layout of the composite image and the specified question-and-answer template, question-and-answer generation processing is performed on the first noun and the second noun to obtain the corresponding question-and-answer pairs; The question-and-answer pair is used as the question-and-answer text.

6. The data processing method based on artificial intelligence according to claim 1, characterized in that, The data generation method based on the target data pair, which involves constructing a corresponding target training dataset from the image and text training dataset according to a preset splicing ratio, specifically includes: Obtain the training target corresponding to the language model, and determine the splicing ratio corresponding to the language model based on the training target; Based on the splicing ratio and the image and text training dataset, a specified number of spliced ​​samples to be generated is determined; Based on the data generation method of the target data pair, the image and text training dataset is processed to generate samples to obtain target splicing samples corresponding to the specified number; Obtain the remaining original samples after processing the image and text training dataset with sample generation; The target spliced ​​sample is combined with the original sample to construct the corresponding target training dataset.

7. The data processing method based on artificial intelligence according to claim 6, characterized in that, The step of determining a specified number of spliced ​​samples to be generated based on the splicing ratio and the image-text training dataset specifically includes: Obtain the total number of samples in the image and text training dataset; Call the preset splicing ratio calculation formula; Based on the splicing ratio calculation formula, the total number of samples and the splicing ratio are solved by algebraic operations to obtain the corresponding solution results; The solution result is used as the specified number of spliced ​​samples to be generated.

8. A data processing device based on artificial intelligence, characterized in that, include: An extraction module is used to randomly extract two image-text pairs from a preset image-text training dataset; wherein, the two image-text pairs include a first image-text pair and a second image-text pair, the first image-text pair includes a first image and a first text description, and the second image-text pair includes a second image and a second text description; The stitching module is used to create a blank canvas based on a preset stitching mode, and to perform image space stitching processing on the first image and the second image based on the blank canvas to obtain a corresponding composite image. The first generation module is used to process the first text description and the second text description based on a preset spatial template library to generate a text description corresponding to the composite image. The second generation module is used to process the first text description and the second text description based on a preset question and answer template library to generate question and answer text corresponding to the composite image; The first construction module is used to generate a corresponding text space description based on the text description and the question-and-answer text, and to construct a corresponding target data pair based on the composite image and the text space description; The second construction module is used to construct the corresponding target training dataset by performing sample construction on the image and text training dataset according to the data generation method of the target data pair and the preset splicing ratio. The processing module is used to train a preset language model based on the target training dataset to obtain a trained target language model, and to perform inference processing on the image data to be processed based on the target language model.

9. A computer device, characterized in that, It includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the steps of the data processing method based on artificial intelligence as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the data processing method based on artificial intelligence as described in any one of claims 1 to 7.

Images