Data augmentation method, system, device, medium and program product based on eigenvector reorganization

Abstract

This invention provides a data augmentation method, system, device, medium, and program product based on feature vector recombination, relating to the fields of deep learning and data augmentation technology. The data augmentation method provided by this invention includes: extracting features from multimodal images from different devices to obtain feature vectors; dividing each feature vector proportionally to obtain task factors and device factors; concatenating the task factors and device factors between different feature vectors to obtain a fused feature vector; and using the feature vectors and the fused feature vector as data samples for downstream tasks. This invention integrates feature decomposition and recombination into an end-to-end process, achieving lightweight augmentation without additional computational resources through random sample pair selection and downstream optimization. The flexibility of this framework makes it applicable to various fields, solving the problem of insufficient coverage of traditional augmentation methods in cross-device scenarios, thereby improving the overall robustness of the model.

Description

Technical Field

[0001] This invention relates to the fields of deep learning and data augmentation technology, and in particular to data augmentation methods, systems, devices, media and program products based on feature vector recombination. Background Technology

[0002] In the fields of machine learning and deep learning, especially in computer vision and medical image analysis, data augmentation techniques are widely used to improve the generalization ability and robustness of models for various tasks. Data augmentation aims to help models better adapt to different input environments by diversifying training data, thereby improving their performance in the real world. Common traditional data augmentation methods include rotation, scaling, flipping, and noise addition, which improve the model's adaptability to input changes to some extent. However, traditional data augmentation methods are difficult to fully realize when dealing with complex tasks and cross-device application scenarios.

[0003] In the field of medical imaging, due to the significant heterogeneity of data generated by different devices and imaging conditions, relying solely on traditional data augmentation methods is insufficient to meet the demands for model generalization capabilities. For example, imaging data for diseases such as cardiomyopathy, pneumonia, and tumors often come from devices from different brands. The differences between these devices in imaging parameters, signal-to-noise ratios, and imaging contrast significantly affect the training performance of the model. Studies have shown that models trained on one brand of device typically perform poorly on data from other brands of devices; this phenomenon is known as the "domain transfer problem."

[0004] Traditional data augmentation methods fail to adequately consider the relationship between device-specific features and image content features (task-related features). Device-related features (such as image noise and imaging style) are often confused with task-related features (such as lesion shape and texture), making models susceptible to interference from device-related features when faced with input from unknown devices, thus affecting overall performance. This problem is particularly prominent in cross-device application scenarios, thus urgently requiring the development of more effective data augmentation methods to address such scenarios.

[0005] In recent years, researchers have begun to explore solutions to these challenges. Two complementary but fundamentally different approaches typically exist: First, a "denoising" strategy, which aims to remove or suppress device-related components from feature representations as much as possible, making the embeddings nearly device-invariant. For example, Domain-Adversarial Neural Networks (DANNs) minimize inter-domain differences through adversarial training, achieving domain invariance of features. However, this strategy has a drawback: it relies excessively on accurate modeling and removal of device noise. Inaccurate modeling or excessive removal can lead to the loss of task-related features, thereby weakening the model's overall expressive power and generalization performance. Second, a "coverage" strategy, which expands the representation coverage of the training domain in the device direction by generating diverse representations of the same content in different device domains at the feature or data level, thereby enhancing the stable representation and consistency of core content features in multi-device scenarios. For example, generative models such as Diffusion Models and Generative Adversarial Networks (GANs) are used to generate sample representations of images on different devices. However, such methods require accompanying image data (i.e., scanned images of the same object on different devices) to train the generative model, which is often difficult to obtain in medical datasets. Furthermore, because they operate at the image level, additional models need to be trained, resulting in high computational time and resource overhead.

[0006] Therefore, there is an urgent need for a new data augmentation method to effectively improve the robustness and generalization ability of models in cross-device scenarios in order to meet the complex challenges in medical image analysis. Summary of the Invention

[0007] In view of the shortcomings of the prior art described above, the purpose of this invention is to provide a data augmentation method, system, device, medium and program product based on feature vector recombination, to solve the technical problem of data interference in cross-device application scenarios.

[0008] To achieve the above and other related objectives, a first aspect of the present invention provides a data augmentation method based on feature vector recombination, comprising: extracting features from multimodal images from different devices to obtain feature vectors; dividing each feature vector proportionally to obtain task factors and device factors; concatenating the task factors and device factors between different feature vectors to obtain a fused feature vector; and using the feature vectors and the fused feature vector as data samples for a downstream task.

[0009] In some embodiments of the first aspect of the present invention, in the step of extracting features from an image to obtain a feature vector, an encoder is used to extract features from the image.

[0010] In some embodiments of the first aspect of the present invention, the training method of the encoder includes: extracting feature vectors of an image using the encoder; dividing each feature vector proportionally to obtain a task factor and a device factor; and supervising the task factor and the device factor respectively to adjust the weights of the encoder during feature extraction.

[0011] In some embodiments of the first aspect of the invention, the task factor uses the classification loss of the downstream task as supervision, and the device factor uses the device classification loss as supervision.

[0012] In some embodiments of the first aspect of the invention, the ratio is used as a hyperparameter and is updated along with the hyperparameters of the downstream task when the downstream task is trained using validation examples.

[0013] In some embodiments of the first aspect of the invention, the ratio between the task factor and the device factor is (7~9):(3~1).

[0014] To achieve the above and other related objectives, a second aspect of the present invention provides a data augmentation system based on feature vector recombination, comprising: a feature acquisition module for extracting features from multimodal images from different devices to obtain feature vectors; a feature partitioning module for partitioning each of the feature vectors proportionally to obtain task factors and device factors; a feature fusion module for concatenating the task factors and device factors between different feature vectors to obtain a fused feature vector; and a data processing module for using the feature vectors and the fused feature vector as data samples for downstream tasks.

[0015] To achieve the above and other related objectives, a third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data augmentation method based on feature vector recombination.

[0016] To achieve the above and other related objectives, a fourth aspect of the present invention provides a computer program product comprising computer program code that, when executed on a computer, causes the computer to implement the data augmentation method based on feature vector reorganization.

[0017] To achieve the above and other related objectives, a fifth aspect of the present invention provides a computer device / apparatus / system / electronic terminal, including a memory, a processor, and a computer program stored in the memory; the processor executes the computer program to implement the data augmentation method based on feature vector reorganization.

[0018] As described above, the data augmentation method, system, apparatus, medium, and program product based on feature vector reorganization of the present invention have the following beneficial effects:

[0019] 1. Controllable Decomposition Mechanism of Feature Vectors: This invention innovatively decomposes the feature vector F into a device factor (F_device) and a task factor (F_task) through dual-head supervised learning, and uses a specified ratio and joint loss optimization to ensure the semantic separation and independence of the two. This decomposition differs from traditional domain adaptation methods (such as adversarial removal in DANNs), providing a structured and interpretable foundation for feature separation, facilitating subsequent reorganization operations, and thus providing an efficient component-based processing method for data augmentation.

[0020] 2. Feature Recombination Exchange Strategy: This invention proposes a vector-level concatenation-based recombination method that generates virtual samples with "same origin, different device" characteristics by exchanging device factors. This strategy simulates multi-domain variation, expanding the diversity of the training set without generating models or real paired data, and significantly improving cross-device generalization ability. Compared with existing style mixing methods (such as statistical mixing in MixStyle), this method focuses more on the explicit exchange of semantic factors, reducing noise introduction and improving enhancement efficiency.

[0021] 3. Integrated Data Augmentation Framework: This invention integrates feature decomposition and recombination into an end-to-end process. Through random sample pair selection and downstream optimization (such as adding F_new to the training set), it achieves lightweight augmentation without additional computational resources. The flexibility of this framework (such as the customizable feature vector partitioning ratio and sample pair selection method) makes it applicable to multiple fields, solving the problem of insufficient coverage of traditional augmentation methods in cross-device scenarios, thereby improving the overall robustness of the model. Attached Figure Description

[0022] Figure 1 This is a flowchart illustrating the data augmentation method based on feature vector reorganization in an embodiment of the present invention.

[0023] Figure 2 This is a schematic diagram illustrating the training and inference of the data augmentation method based on feature vector recombination in an embodiment of the present invention.

[0024] Figure 3 This is a schematic diagram of the encoder training process in an embodiment of the present invention.

[0025] Figure 4 This is a schematic diagram of the structure of the data augmentation system based on feature vector recombination in an embodiment of the present invention.

[0026] Figure 5 This is a schematic diagram of the structure of the computer device in an embodiment of the present invention. Detailed Implementation

[0027] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0028] To address the technical problems described above, this invention introduces a novel method for recombining feature vectors to enhance data, particularly for multimodal images across devices. It should be understood that the method provided in this embodiment is primarily applicable beyond the medical field, but can also be used in industrial inspection, video surveillance, military defense, aerospace, autonomous driving, and other fields; this invention does not limit its application to these areas.

[0029] In one embodiment, such as Figure 1 As shown, a data augmentation method based on feature vector reorganization is provided. This embodiment illustrates the application of this method to a terminal. It is understood that this method can also be applied to a server, and further to a system including both a terminal and a server, and implemented through interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0030] Step 1) Extract features from multimodal images from different devices to obtain feature vectors.

[0031] The dimensions of the multimodal images can be the same or different, but the dimensions of the feature vectors should be the same.

[0032] In practice, different models of medical imaging equipment, i.e., medical imaging equipment from different manufacturers, can be used to acquire medical images, thereby obtaining multimodal medical images. The medical images obtained from different devices are input to the terminal, enabling the terminal to acquire multimodal images. The terminal then performs feature extraction on the multimodal images to obtain the feature vector of each medical image.

[0033] In one embodiment, taking the field of magnetic resonance imaging (MRI) as an example, different models of MRI equipment, i.e., MRI equipment manufactured by different companies, can be used to scan and obtain multimodal MRI images. The MRI images obtained from different MRI equipment are input to the terminal, enabling the terminal to acquire multimodal MRI images. The terminal then performs feature extraction on the multimodal MRI images to obtain the feature vector of each MRI image.

[0034] Step 2) Divide each of the feature vectors proportionally to obtain the task factor and the device factor.

[0035] The task factor and the equipment factor are each part of the feature vector. The division can be linear or non-linear. In linear division, the feature vector can be flattened into a one-dimensional vector and then directly truncated to divide the feature vector into the task factor and the equipment factor.

[0036] The ratio should be the global impact factor, meaning that each feature vector should be divided according to this ratio to ensure that the lengths of task factors and equipment factors are the same after the division.

[0037] In a specific implementation, taking the field of magnetic resonance imaging as an example, the terminal divides the feature vector converted from each magnetic resonance image proportionally to obtain task factors and device factors. The task factors are used to represent image features, while the device factors represent magnetic resonance equipment features.

[0038] Step 3) Concatenate the task factor and device factor between the different feature vectors to obtain the fused feature vector.

[0039] Here, concatenation refers to joining the task factors and device factors from different feature vectors. Since all feature vectors have the same size and use the same partitioning method, concatenation only requires using the same method as partitioning.

[0040] In one specific implementation, taking the field of magnetic resonance as an example, the terminal splices the task factors and device factors between different feature vectors to obtain a fused feature vector.

[0041] Step 4) Use the feature vector and the fused feature vector as data samples for downstream tasks.

[0042] The downstream task can be a classification task.

[0043] In one specific implementation, taking the field of magnetic resonance as an example, the terminal uses the feature vector and the fused feature vector as data samples for downstream tasks, which may be the identification of a certain disease or the classification of multiple diseases.

[0044] The aforementioned data augmentation method extracts features from multimodal images from different devices to obtain feature vectors; divides each feature vector proportionally to obtain task factors and device factors; concatenates the task factors and device factors from different feature vectors to obtain a fused feature vector; and uses the feature vectors and fused feature vectors as data samples for downstream tasks. This method overcomes the shortcomings of traditional augmentation methods and existing generative models, and can improve the robustness and generalization ability of deep learning models in complex tasks and cross-device scenarios. It has wide application value in multiple fields such as medical image analysis and computer vision.

[0045] In embodiments of the present invention, step 1) further includes: extracting features from the image using an encoder. The encoder may employ common encoding methods such as autoencoders, convolutional encoders, or Transformers.

[0046] Specifically, the training method for the encoder includes, as follows: Figure 2 and Figure 3 The following steps are shown:

[0047] Step T1) Use the encoder to extract the feature vector of the image.

[0048] Step T2) Divide each feature vector proportionally to obtain a task factor and a device factor. Note that this ratio is a hyperparameter and can be preset, for example, 7:3. Subsequently, it is updated along with the hyperparameters of the downstream task during training with validation examples. Further, the ratio between the task factor and the device factor is (7~9):(3~1), preferably 7:3.

[0049] Step T3) Supervise the task factor and device factor separately, adjusting the encoder weights during feature extraction. Specifically, the task factor is supervised by the classification loss of the downstream task, and the device factor is supervised by the device classification loss. It is particularly important to note that the encoder weights during image extraction are optimized based on the joint loss of the task factor and device factor, thus ensuring semantic separation and independence between the two. Further, as... Figure 2 and Figure 3 As shown, after the encoder training is completed, its weights are locked. Then, when constructing data samples for downstream tasks, the encoder is directly used to process the feature vectors to expand the data samples for downstream tasks.

[0050] In an embodiment of the present invention, in step 2), a trained encoder is used to divide each feature vector proportionally to obtain task factors and device factors.

[0051] For example, sample pairs A and B from different devices are selected according to preset rules, and their corresponding feature vectors and their decompositions are as follows: and .in, Let the task factor be denoted as sample A. Let the device factor be denoted as sample A. Let the task factor be denoted as sample B. This is represented as the device factor for sample B.

[0052] In an embodiment of the present invention, in step 3), when splicing task factors and equipment factors, each task factor is spliced ​​with all other equipment factors to obtain n.2 The number of data samples, i.e., the number of feature vectors and fused feature vectors, remains at n. 2 This allows for the virtual representation of "different devices with the same features," thus expanding the diversity of the feature space.

[0053] It is understandable that an encoder can be used to concatenate task factors and device factors, and the concatenation rules should be the same as the feature vector splitting rules.

[0054] For example, for sample pair A and B, the device factors between the sample pair are interchanged to obtain a recombined feature vector. , respectively and .in, This is represented as a simulation of the image representation in the feature space when sample A originates from the device containing sample B. This is represented as a simulation of the image representation in the feature space when sample B originates from the same device as sample A.

[0055] In an embodiment of the present invention, in step 4), the feature vector and the fused feature vector are used as data samples for the downstream task. Specifically, the F_new generated in step 3) is added to the training set as an augmentation sample for downstream model optimization.

[0056] To facilitate understanding, the model training process of the feature vector reorganization-based data augmentation method provided in this invention will be fully demonstrated by combining it with downstream tasks:

[0057] Step 1: Use an encoder to extract features from the multimodal image to obtain feature vectors;

[0058] Step 2: Divide the feature vector into device factors and task factors proportionally;

[0059] Step 3: Use the classification loss of the downstream task as the supervision of the task factor and the device classification loss as the supervision of the device factor to adjust the weights of the encoder when extracting features from multimodal images.

[0060] Step 4: Lock the encoder weights and use the encoder to extract features from the multimodal images of the downstream task;

[0061] Step 5: Concatenate the task factor and device factor between different feature vectors to obtain the fused feature vector;

[0062] Step 6: Use the feature vector and the fused feature vector as input to the downstream task, and train the downstream task to obtain a data-enhanced model for the downstream task.

[0063] To facilitate understanding, the model inference process of the feature vector reorganization-based data augmentation method provided by this invention will be fully demonstrated by combining it with downstream tasks:

[0064] Step 1: The encoder extracts features from the input image to obtain feature vectors.

[0065] Step 2: The feature vector is used as input to the downstream task.

[0066] Step 3: Downstream tasks classify the feature vectors.

[0067] Example 1

[0068] This embodiment uses the classification task of cardiac magnetic resonance multi-sequence images as an example to demonstrate the use and effect of feature vector recombination in the feature space, thereby enhancing training data. This example supports the construction of a five-classification model to distinguish five major categories: hypertrophic cardiomyopathy, dilated cardiomyopathy, non-dilated left ventricular cardiomyopathy, restrictive cardiomyopathy, and arrhythmogenic right ventricular cardiomyopathy. By implementing the method of this invention, diverse feature samples are generated, and the robustness and generalization ability of the classification model are significantly improved in complex tasks and cross-device scenarios.

[0069] The first step, dataset construction and preprocessing, involved using cardiac magnetic resonance imaging (MRI) images of 1,000 patients from multiple medical centers, employing various sequences (4CH cine, SAX cine, SAX LGE) covering five categories. During data preprocessing, images acquired from different devices were standardized to ensure data consistency and comparability, thus providing a foundation for subsequent feature extraction and enhancement. The training dataset samples came from GE, Siemens, and Philips devices, while the test dataset samples came from UIH devices. Two samples from the UIH device were incorporated during feature decomposition and reconstruction to assist in feature vector reconstruction.

[0070] The second step is feature decomposition: The input data of each sequence (4CH cine, SAX cine, SAX LGE) are processed by their respective encoders (Video Swin Transformer) to extract feature vectors F. Then, the feature vector F is divided into two parts along the feature dimension according to a preset ratio, serving as the task factor and the device factor, respectively. The task factor F_task is then used in the downstream disease prediction task to predict disease labels through the task classification head (linear layer), using the classification loss of the disease prediction task. As a form of oversight. Among them, For disease labels, Predicting disease probability. Equipment factor. To distinguish the source device of a sample, the device label is predicted through a device classification head (linear layer), using device classification loss. As a form of oversight. Among them, For equipment labels, Predict probabilities for devices. Device factors and task factors are trained synchronously through joint optimization to ensure semantic separation and independence between the two.

[0071] The third step is feature recombination: For each sequence, new feature samples are generated based on the sample exchange factor for each batch. Specifically, random sampling is performed from the batch; for any sample pair A and B, its corresponding feature vector and its decomposition are... and The device factors between sample pairs are interchanged to obtain a recombined feature vector. , respectively and .in That is, it simulates the representation of the image in the feature space when sample A originates from the device where sample B is located; This simulates the representation of an image in the feature space when sample B originates from the same device as sample A. This method generates virtual representations of "same sample, different device," expanding the diversity and coverage of the training data and achieving data augmentation.

[0072] The fourth step is to optimize the downstream model after data augmentation: For each sequence, combine all the original feature vectors F with the newly generated feature vectors. These sequences are used together to train the downstream classifier. Because the training process covers multiple device styles for each sample, the classification model can establish stable disease discrimination boundaries across a wider range of device distributions. Ultimately, the improved performance of the classification model based on each sequence demonstrates that the data augmentation method based on feature vector recombination proposed in this invention can effectively improve the generalization performance of the model in cross-device scenarios.

[0073] Experimental results demonstrate that the feature space enhancement method of this invention achieves significant performance gains in diagnostic experiments for various cardiomyopathy subtypes across different sequences. In the SAX LGE sequence, the overall accuracy of the model reached 0.82, a 20% improvement over the baseline. In the SAX cine and 4CH cine sequences, the accuracy improvements were 10% and 4%, respectively. These results indicate that the feature space enhancement method based on device factor exchange significantly and stably improves generalization performance in the external device domain.

[0074] Since the model training process, inference process, and the entire implementation process have been described in detail in the above embodiments, they will not be repeated here.

[0075] It should be noted that in the embodiments of the present invention, the words "exemplary" or "for example" indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present invention should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of the words "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0076] In this embodiment of the invention, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0077] Figure 4 This is a schematic block diagram of a data augmentation system based on feature vector reorganization provided in an embodiment of the present invention. Figure 4 As shown, the system includes: a feature acquisition module for extracting features from multimodal images from different devices to obtain feature vectors; a feature partitioning module for partitioning each feature vector proportionally to obtain task factors and device factors; a feature fusion module for concatenating task factors and device factors between different feature vectors to obtain fused feature vectors; and a data processing module for using the feature vectors and fused feature vectors as data samples for downstream tasks.

[0078] It should be understood that the specific processes by which each module performs the corresponding steps described above have been detailed in the above method embodiments, and will not be repeated here for the sake of brevity. It should also be understood that the module division in the embodiments of this invention is illustrative and merely a logical functional division; other division methods may exist in actual implementation. Furthermore, the functional modules in the various embodiments of this invention can be integrated into a single processor, exist as separate physical entities, or have two or more modules integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0079] Figure 5 This is a schematic block diagram of a computer device provided in an embodiment of the present invention. Figure 5As shown, the computer device includes at least one processor 601, a memory 602, at least one network interface 603, and a user interface 605. The various components in the device are coupled together via a bus system 604. It is understood that the bus system 604 is used to implement communication between these components. In addition to a data bus, the bus system 604 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 5 The general will label all buses as bus systems.

[0080] The user interface 605 may include a monitor, keyboard, mouse, trackball, clicker, button, touchpad, or touch screen.

[0081] It is understood that memory 602 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM) or programmable read-only memory (PROM), used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM) and synchronous static random access memory (SSRAM). The memories described in the embodiments of this invention are intended to include, but are not limited to, these and any other suitable categories of memory.

[0082] In this embodiment of the invention, the memory 602 is used to store various types of data to support the operation of the electronic terminal 600. Examples of this data include: any executable program for operation on the electronic terminal 600, such as the operating system 6021 and application programs 6022; the operating system 6021 contains various system programs, such as the framework layer, core library layer, driver layer, etc., for implementing various basic services and handling hardware-based tasks. The application program 6022 may contain various applications, such as a media player, browser, etc., for implementing various application services. The data augmentation method based on feature vector reorganization provided in this embodiment of the invention can be included in the application program 6022.

[0083] The methods disclosed in the above embodiments of the present invention can be applied to processor 601, or implemented by processor 601. Processor 601 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 601 or by instructions in the form of software. The processor 601 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 601 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. General-purpose processor 601 may be a microprocessor or any conventional processor, etc. The steps of the accessory optimization method provided in the embodiments of the present invention can be directly reflected as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium, which is located in memory. The processor reads the information in the memory and combines it with its hardware to complete the steps of the aforementioned method.

[0084] In an exemplary embodiment, the electronic terminal 600 may be used by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), or complex programmable logic devices (CPLDs) to perform the aforementioned method.

[0085] According to the method provided in the embodiments of the present invention, the present invention also provides a computer program product, the computer program product comprising: computer program code, which, when executed on a computer, causes the computer to perform... Figures 1 to 2 Data augmentation methods based on feature vector reorganization in any of the embodiments shown.

[0086] According to the method provided in the embodiments of the present invention, the present invention also provides a computer-readable storage medium storing program code, which, when run on a computer, causes the computer to perform the above-described method.

[0087] As used in this specification, the terms "component," "module," "system," etc., are used to refer to computer-related entities, hardware, firmware, combinations of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program, and / or a computer. As illustrated, applications running on computing devices and computing devices can both be components. One or more components may reside in a process and / or an execution thread, and components may be located on a single computer and / or distributed among two or more computers. Furthermore, these components can be executed from various computer-readable media on which various data structures are stored. Components can communicate, for example, via local and / or remote processes based on signals having one or more data packets (e.g., data from two components interacting with another component between a local system, a distributed system, and / or a network, such as the Internet interacting with other systems via signals).

[0088] Those skilled in the art will recognize that the various illustrative logical blocks and steps described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0089] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0090] In the embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0091] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0092] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0093] In the above embodiments, the functions of each functional unit can be implemented entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. A computer program product includes one or more computer instructions (programs). When the computer program instructions (programs) are loaded and executed on a computer, all or part of the flow or function according to the embodiments of the present invention is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., high-density digital video discs, DVDs), or semiconductor media (e.g., solid-state disks, SSDs, etc.).

[0094] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0095] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

[0096] In summary, this invention provides a data augmentation method, system, device, medium, and program product based on feature vector recombination. Therefore, this invention effectively overcomes the various shortcomings of existing technologies and has high industrial applicability.

[0097] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

[0098] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and substitutions can be made without departing from the principles of the present invention, and these improvements and substitutions should also be considered within the scope of protection of the present invention.

Claims

1. A data augmentation method based on feature vector reorganization, characterized in that, The method includes: Feature vectors are obtained by extracting features from multimodal images from different devices; Each of the aforementioned feature vectors is divided proportionally to obtain the task factor and the device factor; By concatenating the task factor and device factor among the different feature vectors, a fused feature vector is obtained; The feature vector and the fused feature vector are used as data samples for downstream tasks.

2. The method according to claim 1, characterized in that, In the process of extracting features from an image to obtain a feature vector, an encoder is used to extract features from the image.

3. The method according to claim 2, characterized in that, The training method for the encoder includes: The encoder is used to extract the feature vector of the image; Each of the aforementioned feature vectors is divided proportionally to obtain the task factor and the device factor; The task factor and device factor are supervised separately, and the weights of the encoder during feature extraction are adjusted.

4. The method according to claim 3, characterized in that, The task factor uses the classification loss of downstream tasks as supervision, and the device factor uses the device classification loss as supervision.

5. The method according to claim 1, characterized in that, The ratio is used as a hyperparameter and is updated along with the hyperparameters of the downstream task when the downstream task is trained using validation examples.

6. The method according to claim 1, characterized in that, The ratio between the task factor and the equipment factor is (7~9):(3~1).

7. A data augmentation system based on feature vector recombination, characterized in that, include: The feature acquisition module is used to extract features from multimodal images from different devices to obtain feature vectors; The feature segmentation module is used to divide each of the feature vectors proportionally to obtain task factors and device factors; The feature fusion module is used to concatenate the task factor and device factor between different feature vectors to obtain a fused feature vector. The data processing module is used to use the feature vector and the fused feature vector as data samples for downstream tasks.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the data augmentation method based on feature vector reorganization as described in any one of claims 1 to 6.

9. A computer program product, characterized in that, The computer program product includes computer program code that, when run on a computer, causes the computer to implement the data augmentation method based on feature vector reorganization as described in any one of claims 1 to 6.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the data augmentation method based on feature vector reorganization as described in any one of claims 1 to 6.

Images