Multimodal data expansion method, system, medium, computer equipment and terminal

Abstract

The present invention belongs to the field of computer data processing technology, and discloses a multimodal data expansion method, system, medium, computer equipment and terminal. Without changing the semantic information of any modal data, the image features are disturbed by adjusting the image content in the receptive field of the target detection model, thereby automatically expanding the data, achieving the effect of reducing labor costs and improving data expansion efficiency. The rich data provided can improve the performance of downstream tasks. The multimodal data expansion method provided by the present invention performs data expansion by expanding image features, and can automatically expand the data without changing the semantic information of any modal data. Therefore, the present invention does not change the semantic information of any modality in the multimodal training data, and the data expansion effect is good. At the same time, the present invention can automatically expand the data, with low labor costs and high data expansion efficiency.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer data processing, and in particular, relates to a multimodal data expansion method, system, medium, computer equipment and terminal. Background Art

[0002] At present, with the development of multimedia and Internet technology, the use of multimodal information to describe events and things has become a common phenomenon, for example, combining image and text modalities for news reporting, combining video and audio modalities for short video production, and so on. Generally speaking, there is a correlation between data of different modalities that appear at the same time, and analyzing this correlation is of great significance for data mining and data protection. At present, in the field of multimodal machine learning, related research directions include: Image Captioning, Cross-modal Retrieval, Visual Question Answering, etc., which provide open source data sets, which provide support for the technical verification of multimodal machine learning. In order to achieve better machine learning results and improve the usability of the model in actual scenarios, it is often necessary to expand the data and train the model on a richer data set.

[0003] One option for data augmentation is to manually collect and annotate data, but this is time-consuming, labor-intensive and inefficient. Existing technologies prefer to automatically augment data. For image modalities, classic automatic data augmentation methods include image flipping, chromaticity adjustment, random erasure, etc. For text modalities, there are synonym replacement, back translation, random noise injection, etc. However, in the context of multimodal machine learning, these traditional methods are difficult to meet the needs of data augmentation. Take the image description task as an example. The goal of this task is to generate a corresponding natural language description for the image. Each sample in the training set consists of a picture and a text description. If the image chromaticity adjustment method is used to augment the data, the "color" semantics of the augmented image will be inconsistent with the description in the text segment. For example, a "red" apple may become a "gray-green" apple after automatic chromaticity adjustment. Similarly, image flipping may cause changes in "orientation" semantics, and random erasure may cause the loss of image semantics. These semantic changes are currently difficult to automatically and accurately correct in text descriptions. Therefore, it is urgent to design a new data augmentation method.

[0004] Through the above analysis, the problems and defects of the prior art are as follows:

[0005] (1) The traditional method of manually collecting and labeling data is time-consuming, labor-intensive and inefficient.

[0006] (2) In the context of multimodal machine learning, traditional methods are difficult to meet the needs of data expansion.

[0007] (3) Existing data augmentation methods may lead to the loss of image semantics, and these semantic changes are currently difficult to automatically and accurately correct in text descriptions.

[0008] The difficulty in solving the above problems and defects is as follows: (1) It requires a lot of labor costs, which is difficult. There is currently no unified solution to the above problems (2) and (3), which is difficult to solve.

[0009] The significance of solving the above problems and defects is: (1) reducing the labor cost of data expansion and improving the efficiency of data expansion. (2) providing a feasible solution for the needs of multimodal data expansion. (3) it can be applied to multimodal tasks such as cross-modal retrieval, visual question answering, image description, etc., making its training samples richer and improving the training effect. Summary of the invention

[0010] In view of the problems existing in the prior art, the present invention provides a multimodal data expansion method, system, medium, computer device and terminal.

[0011] The present invention is implemented as follows: a multimodal data expansion method, the multimodal data expansion method comprising: expanding image features Implement data expansion; extract When the target is detected, the convolutional neural network-based target detection model is used to extract the target. By disturbing the content of the image in the receptive field and changing the position of the target detection frame, the Make disturbances.

[0012] Furthermore, the multimodal data expansion method comprises the following steps:

[0013] Step 1: randomly select k images each time and splice the images in the data set; by splicing images, the image content in the receptive field is changed, laying the foundation for perturbing image features and expanding data.

[0014] Step 2: Perform target detection on the stitched large image to obtain a set of detection frames. This is a conventional step for extracting image features based on a target detection model. By observing and comparing the detection frames obtained in this step with those obtained before stitching, it is found that the positions and sizes of the frames are different, which provides a basis for data expansion.

[0015] Step 3: Group the detection frames in the detection frame set, and group the detection frames that originally belonged to one image into one group. For one spliced ​​image, a total of k groups of detection frames are obtained;

[0016] Step 4: Extract features from the image regions corresponding to each group of detection boxes, and combine them with the text descriptions of the corresponding original images to form a new pair of training samples. For each stitched image, a total of k pairs of new training samples are obtained.

[0017] In Steps 3 and 4 of the present invention: For any sub-image in the puzzle (i.e., an original image before stitching), it is ensured that the features obtained through expansion do not contain the features of other images in the puzzle, thus ensuring that data expansion does not change the semantics of the original image and playing a positive role in high-quality expansion. In Step 4, each stitching can obtain k pairs of new training samples, playing a positive role in efficient expansion.

[0018] Furthermore, after the images are stitched in Step 1, some image contents within the receptive field change.

[0019] Furthermore, the image stitching in Step 1 includes:

[0020] Let the set of all images in the multi-modal dataset D be I = {I 1 , I 2 ,..., I n}. For each I i , randomly select k non-repeating images I K ∈I and stitch them into a large image including:

[0021] (1) In a complete data expansion process, for each I i , k remains the same during stitching;

[0022] (2) k is an integer, 2 ≤ k ≤ 9;

[0023] (3) The arrangement principle for stitching is to make the aspect ratio of the stitched image as small as possible. When k = 2, the two images are stitched left and right or up and down. When k = 4, the images are stitched in a "field" shape. When k = 6, the images are stitched in three rows and two columns or two rows and three columns. When k = 9, the images are arranged in a "nine-square grid" shape. For other values, they are stitched in the way closest to the described arrangement;

[0024] (4) The aspect ratio of any image is not changed during stitching. After stitching according to the above layout, the unaligned parts are filled with 0 values.

[0025] Furthermore, the obtaining of the set of detection boxes in Step 2 includes:

[0026] Obtain detection boxes based on the object detection model, with the following requirements:

[0027] (1) According to k and the splicing plate type, proportionally enlarge the pixel limits of the long side and the short side of the target detection model. When k = 2 and taking left - right splicing, the long - side threshold is taken as 2 times the original threshold, and the short - side threshold remains unchanged. When k = 4, in a "field" - shaped splicing, both the long - side and short - side thresholds are 2 times the original threshold. When k = 9, in a "nine - grid" - shaped splicing, both the long - side and short - side thresholds are 3 times the original threshold;

[0028] (2) Input into the target detection model to obtain a set of detection - box positions with a total of m * k detection boxes;

[0029] (3) Group the detection boxes according to the pictures they belonged to before splicing to obtain where each where p is the confidence of the detection box;

[0030] (4) For with a length greater than m, sort the detection boxes according to their confidence and take the top m boxes with the highest confidence. Otherwise, fill them with P = ((0,0),(0,0),0);

[0031] (5) Since the detection boxes are distributed on and the obtained coordinates are all with the upper - left corner as the origin; for each K in the set I according to the position and coordinate set on correct the coordinates to the coordinates with the upper - left corner as the origin, so as to be consistent with the data distribution of the original data set. The corrected coordinates are represented as The obtained k groups of coordinate sets are represented as

[0032] Further, the acquisition of the new training samples in step (3) includes:

[0033] For each i in O and its corresponding take the corresponding - area picture, input it into the feature extraction model to obtain a new feature Then is a pair of new training samples. In this way, k new samples are obtained through one - time splicing. If the size of the picture set in the original data set is n, then after one complete data - augmentation process, n * k new training samples are obtained.

[0034] Another object of the present invention is to provide a multi - modal data - augmentation system applying the multi - modal data - augmentation method described above. The multi - modal data - augmentation system includes:

[0035] The image stitching module is used to stitch the images in the data set by randomly selecting k images each time;

[0036] The detection frame set acquisition module is used to perform target detection on the spliced ​​large image to obtain the detection frame set;

[0037] The detection frame grouping module is used to group the detection frames in the detection frame set, and group the detection frames originally belonging to one image into one group. For one spliced ​​image, a total of k groups of detection frames are obtained;

[0038] The training sample acquisition module is used to extract features from the image area corresponding to each group of detection frames, and combine them with the text description of the corresponding original image to form a pair of new training samples. For each spliced ​​image, a total of k pairs of new training samples are obtained.

[0039] Another object of the present invention is to provide a computer device, the computer device comprising a memory and a processor, the memory storing a computer program, and when the computer program is executed by the processor, the processor performs the following steps:

[0040] By expanding Implement data expansion; extract When the target detection model based on convolutional neural network mentions the features related to the size of the receptive field, the position of the target detection box calibration, and the content of the picture, the size of the receptive field remains unchanged when the network structure remains unchanged, and the extracted features are disturbed by changing the content of the picture in the receptive field or the position of the target detection box calibration.

[0041] Another object of the present invention is to provide a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor executes the following steps:

[0042] By expanding Implement data expansion; extract When the target detection model based on convolutional neural network mentions the features related to the size of the receptive field, the position of the target detection box calibration, and the content of the picture, the size of the receptive field remains unchanged when the network structure remains unchanged, and the extracted features are disturbed by changing the content of the picture in the receptive field or the position of the target detection box calibration.

[0043] Another object of the present invention is to provide an information data processing terminal, which is used to implement the multimodal data expansion system.

[0044] Combining all the above technical solutions, the advantages and positive effects of the present invention are as follows: the multimodal data expansion method provided by the present invention performs data expansion by expanding image features, and can automatically perform data expansion without changing the semantic information of any modal data. Therefore, the present invention does not change the semantic information of any modality in the multimodal training data, and the data expansion effect is good; the present invention can automatically perform data expansion, with low labor costs and high data expansion efficiency.

[0045] The present invention has been verified on cross-modal retrieval, visual question answering, and image description tasks, and can enrich training samples and improve training effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0047] Figure 1 It is a flow chart of a multimodal data expansion method provided by an embodiment of the present invention.

[0048] Figure 2 is a structural block diagram of a multimodal data expansion system provided by an embodiment of the present invention;

[0049] In the figure: 1. Image stitching module; 2. Detection frame set acquisition module; 3. Detection frame grouping module; 4. Training sample acquisition module.

[0050] FIG. 3 is a schematic diagram showing a comparison of images before and after splicing provided by an embodiment of the present invention.

[0051] Figure 3A is the image before splicing provided by the embodiment of the present invention. 190141 Schematic diagram of .

[0052] Figure 3B is the image before splicing provided by the embodiment of the present invention. 202099 Schematic diagram of .

[0053] Figure 3C is the spliced ​​picture provided by the embodiment of the present invention Schematic diagram of .

[0054] Figure 4 Schematic diagram of the detection frame position of the spliced ​​image provided by an embodiment of the present invention.

[0055] FIG. 5 is a schematic diagram of the position of the detection frame of the image before stitching provided by an embodiment of the present invention.

[0056] Figure 5A is the image before splicing provided by the embodiment of the present invention. 190141 Schematic diagram of the detection box position.

[0057] Figure 5B is the image before splicing provided by the embodiment of the present invention. 202099 Schematic diagram of the detection box position. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0059] In view of the problems existing in the prior art, the present invention provides a multimodal data expansion method, system, medium, computer device and terminal. The present invention is described in detail below with reference to the accompanying drawings.

[0060] like Figure 1 As shown, the multimodal data expansion method provided by the embodiment of the present invention includes the following steps:

[0061] S101, randomly select k pictures each time and splice the pictures in the existing data set;

[0062] S102, performing target detection on the spliced ​​large image to obtain a set of detection frames;

[0063] S103, grouping the detection frames in the detection frame set, grouping the detection frames originally belonging to one image into one group, and obtaining k groups of detection frames for one spliced ​​image;

[0064] S104, extracting features from the image area corresponding to each group of detection frames, and combining them with the text description of the corresponding original image to form a pair of new training samples.

[0065] like Figure 2 As shown, the multimodal data expansion system provided by the embodiment of the present invention includes:

[0066] Image stitching module 1, used to stitch the images in the data set by randomly selecting k images each time;

[0067] The detection frame set acquisition module 2 is used to perform target detection on the spliced ​​large image to obtain a detection frame set;

[0068] The detection frame grouping module 3 is used to group the detection frames in the detection frame set, and group the detection frames originally belonging to one image into one group. For one spliced ​​image, a total of k groups of detection frames are obtained;

[0069] The training sample acquisition module 4 is used to extract features from the image area corresponding to each group of detection frames, and combine them with the text description of the corresponding original image to form a pair of new training samples. For each spliced ​​image, a total of k pairs of new training samples are obtained.

[0070] The technical solution of the present invention is further described below in conjunction with specific embodiments.

[0071] Example 1

[0072] In view of the problems existing in the prior art, the present invention provides a multimodal data expansion method, which can automatically expand data without changing the semantic information of any modal data.

[0073] 1. Project Description

[0074] Suppose there is a multimodal dataset D = {(I 1 ,T 1 ),(I 2 ,T 2 ),...,(I n ,T n )}, where I i It's a picture, T i is a text corresponding to the picture, (I i , T i ) form a pair of samples. There are n pairs of samples in the data set. For such data, the general process is to extract I i Features and T i Features Then, based on the multimodal machine learning model and modeling the relationship between Constitute a pair of training samples. In particular, extract It is divided into two steps. The first step is to use the convolutional neural network target detection model from I i All target objects are detected in the image, and the locations where the targets appear are represented as a set of size m O = {P 1 ,P 2 ,...P m}, where P j =((w 0 ,h 0 ),(w 1 ,h 1 )), P j is the coordinate of a detection box, which identifies the position of a target object in the image, with the upper left corner of the image as the coordinate origin. (w 0 ,h 0 ) is the coordinate of the upper left corner of the detection box, (w 1 ,h1 ) is the coordinate of the lower right corner, w represents the width (long side), and h represents the height (short side). The second step is to j , extract it in I i The feature vector f of the corresponding region in j ,get

[0075] The present invention expands To achieve data expansion. When the target detection model based on the convolutional neural network mentions the features related to the size of the receptive field, the position of the target detection frame calibration, and the content of the picture, the size of the receptive field remains unchanged when the network structure remains unchanged, then the extracted features can be disturbed by changing the content of the picture in the receptive field, or changing the position of the target detection frame calibration. The present invention hopes to efficiently expand the data without changing the semantics of the image. The proposed idea is: the first step is to splice the pictures in the existing data set, and randomly select k pictures for splicing each time. Through this step, part of the image content in the receptive field changes. Then, target detection is performed on the spliced ​​large picture to obtain a detection frame set. In this step, the target detection frame may change. Then, the detection frames in the detection frame set are grouped, and the detection frames originally belonging to one picture are divided into one group. For a spliced ​​picture, a total of k groups of detection frames are obtained. Through grouping, each group of detection frames excludes the influence of the detection frames that do not belong to this picture on the semantics of the picture. Finally, feature extraction is performed on the picture area corresponding to each group of detection frames, and combined with the text description of the corresponding original picture to form a pair of new training samples. The specific process is described as follows:

[0076] 1. Image stitching

[0077] Suppose the set of all images in the multimodal dataset D is I = {I 1 ,I 2 ,...,I n}, for each I i , randomly select k non-repeated pictures I K ∈I, spliced ​​into a large picture The present invention requires:

[0078] 1) In a complete data expansion process, for each I i , k remains consistent during splicing;

[0079] 2) k is an integer, 2≤k≤9;

[0080] 3) The splicing arrangement principle is to make the aspect ratio of the spliced picture as small as possible. When k = 2, the two pictures can be spliced horizontally or vertically. When k = 4, the pictures are spliced in a "field" shape. When k = 6, the pictures are spliced in three rows and two columns or two rows and three columns. When k = 9, the pictures are arranged in a "nine-square grid" shape. For other values, the splicing is performed in the way closest to the above several arrangements;

[0081] 4) When splicing, the aspect ratio of any picture is not changed. After splicing according to the above layout, the unaligned parts are filled with 0 values.

[0082] 2. Obtain the set of detection frames

[0083] Based on the object detection model, obtain the detection frames. The present invention requires:

[0084] 1) According to k and the splicing layout, proportionally increase the pixel limits of the long side and the short side of the object detection model. For example, when k = 2 and horizontal splicing is selected, the long side threshold is taken as 2 times the original threshold, and the short side threshold remains unchanged. When k = 4 and the "field" splicing is performed, both the long side and the short side thresholds should be 2 times the original threshold. When k = 9 and the "nine-square grid" splicing is performed, both the long side and the short side thresholds should be 3 times the original threshold;

[0085] 2) Input into the object detection model to obtain the set of detection frame positions with a total of m * k detection frames;

[0086] 3) Group the detection frames according to the pictures they belonged to before splicing to obtain where each where p is the confidence of the detection frame;

[0087] 4) Since the detection frames are distributed on , the lengths of the grouped may not be uniform. For with a length greater than m, sort the detection frames according to the confidence and take the top m frames with the highest confidence. Otherwise, fill it with P = ((0,0),(0,0),0);

[0088] 5) Since the detection frames are distributed on , the obtained coordinates are all based on with the upper left corner as the origin. Therefore, for each K in the set I , it is necessary to correct the coordinates to the coordinates with the upper left corner as the origin according to its position on and the coordinate set so as to be consistent with the data distribution of the original data set. The corrected coordinates are represented as The obtained k groups of coordinate sets are represented as

[0089] 3. Get new training samples

[0090] For O i Each of The corresponding Pick Corresponding to the area picture, input the feature extraction model to obtain new features but is a new pair of training samples.

[0091] Through the above method, k new samples can be obtained after one splicing. If the size of the picture set in the original data set is n, then after a complete data expansion process, n*k new training samples can be obtained, which efficiently expands the data without changing the semantics of any modality data.

[0092] Example 2

[0093] This embodiment describes the implementation process of a splicing. Taking the picture set I in the "COCO Caption train2014" data set as an example, k=2, m=10 is taken, that is, 2 pictures are spliced, and 10 features of the target object are detected in each picture.

[0094] 1. Image stitching

[0095] The picture I with the number 000000190141 in I 190141 For example, randomly take a picture set {I 190141 ,I 202099}, for k=2, this embodiment adopts the left-right splicing method to splice the image The stitching does not change the aspect ratio of the two images. 190141 The resolution is 640*423, I 202099 The resolution is 640*480. 190141 and I 202099 The widths are different, and the unaligned parts are filled with 0 values ​​when splicing. The resolution is 1280*480. Figure 3 shows the pictures before and after stitching.

[0096] 2. Get the detection box set

[0097] This embodiment uses the FasterRCNN target detection model to obtain the detection frame.

[0098] 1) The original default long side resolution threshold of the model is 1333, and the short side threshold is 800. According to the number and layout of the stitched pictures, this embodiment enlarges the long side threshold to 2666, and the short side threshold remains unchanged.

[0099] 2) Input the model and take the top 20 detection boxes with the highest confidence. Figure 4 The locations of these detection boxes are shown.

[0100] 3) Group the detection frames according to the pictures they belong to before stitching. In this example, the intersection ratio of the detection frame area and the original picture is calculated to automatically determine whether a detection frame belongs to a certain picture. The calculation method is as follows:

[0101]

[0102]

[0103] 4) In this embodiment, after grouping, Each group has exactly 10 detection boxes, without truncation or padding.

[0104] 5) For Figure I 190141 , it is The coordinates in are (0,0,640,423), which is the same as The coordinate origin of the image coincides with that of the image, and there is no need to correct the detection frame belonging to the image. The coordinate set of the detection frame of the image is For Figure I 202099 , it is The coordinates in are (640, 0, 1280, 480), and the origin of the coordinates is If they do not overlap, the detection frames belonging to the image are corrected to coordinates with the upper left corner of the image as the origin, and the detection frame set is obtained.

[0105] 3. Get new training samples

[0106] Based on FasterRCNN, extract I 190141 correspond Features of the detection box Get new samples Extraction I 202099 correspond Features of the detection box Get new samples

[0107] After the splicing process in this example, two new samples are obtained. Figure 5 shows the image I before splicing. 190141 and I 202099 The positions of the first 10 detection boxes in the comparison Figure 4, the positions of the detection frames of the corresponding pictures are different, and it can be inferred that the features obtained according to these areas are also different. In other words, through the process of the present invention, richer features can be obtained. A complete data expansion requires each sample in the original data set to perform the process shown in this example. After a complete data expansion, n*k new samples can be obtained. The present invention can be applied to multimodal research directions of image-text modalities such as cross-modal retrieval, visual question answering, and image description. Specific application scenarios include: e-commerce platforms retrieving products by text, automatic review of network images based on image descriptions, etc.

[0108] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When the use is implemented in whole or in part in the form of a computer program product, the computer program product includes one or more computer instructions. When the computer program instructions are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL) or wireless (e.g., infrared, wireless, microwave, etc.) mode) to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0109] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with the technical field within the technical scope disclosed by the present invention and within the spirit and principle of the present invention should be covered by the protection scope of the present invention.

Claims

1. A multimodal data expansion method, characterized in that: The multimodal data expansion method expands image features Implement data expansion; extract When the target is detected, the convolutional neural network-based target detection model is used to extract the target. By disturbing the content of the image in the receptive field and changing the position of the target detection frame, the To disturb; The multimodal data expansion method comprises the following steps: Step 1: randomly select k pictures each time and splice the pictures in the data set; Step 2: Perform target detection on the spliced ​​large image to obtain a set of detection frames; Step 3: Group the detection frames in the detection frame set, and group the detection frames that originally belonged to one image into one group. For one spliced ​​image, a total of k groups of detection frames are obtained; Step 4: Extract features from the image area corresponding to each set of detection frames and combine them with the text description of the corresponding original image to form a pair of new training samples. For each spliced ​​image, a total of k pairs of new training samples are obtained. After the images in step 1 are stitched together, part of the image content in the receptive field changes and the position of the target detection frame changes; The image stitching in step 1 includes: the set of all images in the multimodal dataset D is I = {I1, I2, ..., I n }, for each I i , randomly select k non-repeated pictures Stitching into one large picture include: (1) In a complete data expansion process, for each I i , k remains consistent during splicing; (2) k is an integer, 2≤k≤9; (3) The principle of splicing arrangement is to make the aspect ratio of the spliced ​​pictures as small as possible. When k=2, the two pictures are spliced ​​left and right or up and down. When k=4, the pictures are spliced ​​in a field shape. When k=6, the pictures are spliced ​​in three rows and two columns or two rows and three columns. When k=9, the pictures are arranged in a nine-square grid shape. Other values ​​are spliced ​​in a way that is closest to the above arrangement. (4) The aspect ratio of any image is not changed during stitching. After the images are stitched together, any unaligned parts are filled with a value of 0.

2. The multimodal data expansion method according to claim 1, characterized in that: The step 2 of obtaining the detection frame set includes: Obtain the detection frame based on the target detection model. The requirements are as follows: (1) According to k and the splicing pattern, the pixel limits of the long and short sides of the target detection model are proportionally enlarged. When k = 2, the left and right splicing is used, the long side threshold is twice the original threshold, and the short side threshold remains unchanged. When k = 4, the splicing is in the form of a square, and the long and short side thresholds are both twice the original thresholds. When k = 9, the splicing is in the form of a nine-square grid, and the long and short side thresholds are both 3 times the original thresholds. (2) Input the target detection model to get the detection box position set There are m*k detection frames in total; (3) Group the detection frames according to the pictures they belong to before splicing, and get Each of these Where p is the confidence of the detection box; (4) For If the length is greater than m, sort the detection boxes by confidence and take the first m boxes with the highest confidence. Otherwise, fill them with P = ((0,0), (0,0), 0); (5) Since the detection boxes are distributed The coordinates obtained are The upper left corner is the origin; for set I K Each of According to The position and coordinate set on Correct the coordinates to the coordinates with the upper left corner as the origin, so as to keep consistent with the data distribution of the original data set. The coordinates are expressed as The obtained k sets of coordinates are expressed as 3. The multimodal data expansion method according to claim 1, characterized in that: The acquisition of new training samples in step 3 includes: i Each of The corresponding Pick Corresponding to the area picture, input the feature extraction model to obtain new features but is a pair of new training samples; through this method, k new samples are obtained after one splicing. If the size of the picture set in the original data set is n, then after a complete data expansion process, n*k new training samples are obtained.

4. A multimodal data expansion system for implementing the multimodal data expansion method according to any one of claims 1 to 3, characterized in that: The multimodal data expansion system comprises: The image stitching module is used to stitch the images in the data set by randomly selecting k images each time; The detection frame set acquisition module is used to perform target detection on the spliced ​​large image to obtain the detection frame set; The detection frame grouping module is used to group the detection frames in the detection frame set, and group the detection frames originally belonging to one image into one group. For one spliced ​​image, a total of k groups of detection frames are obtained; The training sample acquisition module is used to extract features from the image area corresponding to each group of detection frames, and combine them with the text description of the corresponding original image to form a pair of new training samples. For each spliced ​​image, a total of k pairs of new training samples are obtained.

5. A computer device, characterized in that: The computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the step of enriching the multimodal data according to any one of claims 1 to 3, by expanding F Ii Implement data expansion; expand F Ii This is achieved by changing the content of the image within the receptive field and the position of the target detection frame calibration.

6. A computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor executes the step of enriching the multimodal data according to any one of claims 1 to 3, by expanding F Ii Implement data expansion; expand F Ii This is achieved by changing the content of the image within the receptive field and the position of the target detection frame calibration.

7. An information data processing terminal, characterized in that: The information data processing terminal is used to implement the multimodal data expansion system as claimed in claim 4.

Images