Synthetic data generation method, apparatus and device, and readable storage medium

Abstract

The invention provides a synthetic data generation method and device, equipment and a readable storage medium, and the method comprises the steps: generating an initial image feature according to the demand data of a user, obtaining the image feature of each object instance from the initial image feature, and embedding the class information of each object instance into the image feature of the region where the object instance is located, linear transformation and a self-attention mechanism are combined to obtain self-attention output of the image features added with the category information, and synthetic data are generated through decoding processing, so that different categories of object instances can be distinguished in the synthetic data generation process, feature representation changes of the object instances can be accurately controlled, a user can adjust corresponding scene elements according to requirements, and the user experience is improved. Therefore, the high controllability of the scene in the synthetic data generation process is realized.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method, device, equipment and readable storage medium for generating synthetic data. Background Art

[0002] At present, the training data required for the development of the autonomous driving field and embodied intelligence industry usually comes from real data collected from real physical environments and virtual environment simulation data generated by data twin simulation technology. Among them, the collection cost of real data is high and is restricted by actual conditions, which limits its wide application. The simulation data generated by data twin simulation technology can make up for the problem of insufficient real data to a certain extent.

[0003] However, the current generation process of the simulation data makes it difficult to precisely control some static or dynamic elements without changing the overall semantics of the scene. The uncontrollability of the generation of the synthetic data production scene affects the diversity and quality of the synthetic data. For example, for static environmental elements such as road type, weather conditions, time changes, and dynamic traffic elements such as vehicle type and movement trajectory, existing simulation technology is difficult to achieve precise adjustment and diversified combination of these elements while maintaining scene coherence and consistency. Summary of the invention

[0004] In view of this, in order to solve the above technical problems, the present application provides a synthetic data generation method, device, equipment and readable storage medium.

[0005] Specifically, the present application is implemented through the following technical solutions:

[0006] According to a first aspect of an embodiment of the present application, a method for generating synthetic data is provided, the method comprising:

[0007] Obtaining initial image features corresponding to user demand data;

[0008] Acquire a first image feature of a first subregion where each object instance in the initial image feature is located, and add a learnable category hint of the object instance to the first image feature to obtain a second image feature;

[0009] Performing a linear transformation on the second image feature according to a preconfigured first weight matrix, and applying a self-attention mechanism to obtain a self-attention output of the first sub-region;

[0010] The self-attention output of the first sub-region is decoded according to task requirements to generate target synthetic data.

[0011] Optionally, the method further comprises:

[0012] Acquire a third image feature of a second sub-region that does not contain any object instance in the initial image feature, and add a learnable empty instance hint to the third image feature to obtain a fourth image feature;

[0013] Performing a linear transformation on the fourth image feature according to a preconfigured second weight matrix, and applying the self-attention mechanism to obtain a self-attention output of the second sub-region;

[0014] The decoding process of the self-attention output according to the task requirements includes:

[0015] According to the task requirements, the self-attention outputs of the first sub-region and the second sub-region are decoded.

[0016] Optionally, the decoding process of the self-attention outputs of the first sub-region and the second sub-region according to the task requirement includes:

[0017] Merging the self-attention outputs of the first sub-region and the second sub-region according to the mask information of the first sub-region and the second sub-region in the initial image feature to obtain a corresponding merged attention output;

[0018] The combined attention output is decoded according to the task requirements.

[0019] Optionally, the method further comprises:

[0020] Obtaining a bounding box of the object instance, and encoding the bounding box and the category of the object instance to generate a hidden vector;

[0021] Integrating the hidden vector with the initial image feature through a gating mechanism and a self-attention mechanism to construct a merged image feature;

[0022] The decoding process of the self-attention output of the first sub-region according to the task requirement includes:

[0023] According to the task requirements, the self-attention output of the first sub-region and the merged image features are decoded.

[0024] Optionally, encoding the bounding box and the category of the object instance to generate a hidden vector includes:

[0025] Converting the position information of the bounding box of the object instance into a first representation vector in a high-dimensional feature space, and converting the category of the object instance into a second representation vector;

[0026] Connecting the first representation vector and the second representation vector to obtain a combined vector;

[0027] The combination vector is mapped to a latent space using a nonlinear transformation of a neural network to generate the latent vector.

[0028] Optionally, the step of integrating the hidden vector with the initial image feature through a gating mechanism and a self-attention mechanism to construct a merged image feature includes:

[0029] Performing feature connection between the hidden vector and the initial image feature to obtain a first feature representation;

[0030] Performing a linear transformation on the first feature representation according to a preset third weight matrix, and applying the self-attention mechanism to obtain a self-attention output of the first feature representation;

[0031] A visual tag selection operation and a gating mechanism are applied to the self-attention output of the first feature representation to generate the merged image feature; the visual tag selection operation is used to select the tag most relevant to the task requirement from the self-attention output of the first feature representation.

[0032] Optionally, applying a visual marker selection operation and a gating mechanism to the self-attention output of the first feature representation to generate the merged image feature comprises:

[0033] applying a visual marker selection operation to the self-attention output of the first feature representation to obtain a second feature representation;

[0034] A gating mechanism is applied to the second feature representation, and the corresponding gating mechanism output is added to the initial image feature to generate the merged image feature; the gating mechanism adjusts the contribution of the second feature representation to the initial image feature through a learnable gating parameter and a hyperbolic tangent function.

[0035] Optionally, performing a linear transformation on the first feature representation according to a preset third weight matrix, and applying the self-attention mechanism to obtain a self-attention output of the first feature representation includes:

[0036] Obtaining a task-adaptive embedding; the task-adaptive embedding comprising a plurality of learnable embedded task-adaptive cues of the same dimension as the initial image features;

[0037] Performing feature concatenation on the first feature representation and the task adaptive embedding along the spatial dimension to obtain a merged feature representation;

[0038] Using a third weight matrix for generating a query vector, linearly transforming the first feature representation to generate a first query vector, and using a third weight matrix for generating a key vector and a value vector, linearly transforming the merged feature representation to generate a first key vector and a first value vector;

[0039] A self-attention mechanism is applied to the first query vector, the first key vector, and the first value vector to obtain a self-attention output of the first feature representation.

[0040] Optionally, the method further comprises:

[0041] Obtaining a conditional embedding associated with the task requirement; the conditional embedding includes an embedding vector obtained by encoding specific generation condition information;

[0042] The conditional embedding and the merged feature representation are feature-concatenated along a spatial dimension to obtain an updated merged feature representation.

[0043] Optionally, performing a linear transformation on the second image feature according to a preconfigured first weight matrix and applying a self-attention mechanism to obtain a self-attention output of the first sub-region includes:

[0044] Performing a linear transformation on the second image feature using weight matrices for generating a query vector, a key vector, and a value vector, respectively, to generate a target query, a target key, and a target value for the object instance;

[0045] Apply the self-attention mechanism to the target query, target key, and target value to obtain a self-attention output of the first sub-region.

[0046] Optionally, decoding the self-attention output of the first sub-region according to task requirements to generate target synthetic data includes:

[0047] Determine a target decoder for achieving the task requirement from a plurality of pre-set decoder architectures;

[0048] The self-attention output of the first sub-region is input into the target decoder to obtain the target synthetic data output by the target decoder.

[0049] According to a second aspect of an embodiment of the present application, a synthetic data generating device is provided, the device comprising:

[0050] An initial image feature acquisition module is used to acquire initial image features corresponding to user demand data;

[0051] an instance category hint adding module, configured to obtain a first image feature of a first sub-region where each object instance is located in the initial image feature, and add a learnable category hint of the object instance to the first image feature to obtain a second image feature;

[0052] an instance attention output module, configured to perform a linear transformation on the second image feature according to a preconfigured first weight matrix, and apply a self-attention mechanism to obtain a self-attention output of the first sub-region;

[0053] The synthetic data generation module is used to decode the self-attention output of the first sub-area according to task requirements to generate target synthetic data.

[0054] Optionally, the device further comprises:

[0055] Acquire a third image feature of a second sub-region that does not contain any object instance in the initial image feature, and add a learnable empty instance hint to the third image feature to obtain a fourth image feature;

[0056] Performing a linear transformation on the fourth image feature according to a preconfigured second weight matrix, and applying the self-attention mechanism to obtain a self-attention output of the second sub-region;

[0057] The synthetic data generation module is specifically used to decode the self-attention outputs of the first sub-region and the second sub-region according to the task requirements.

[0058] Optionally, when the synthetic data generation module is used to decode the self-attention outputs of the first sub-region and the second sub-region according to the task requirements, it includes:

[0059] According to the mask information of the first sub-region and the second sub-region in the initial image features, the self-attention outputs of the first sub-region and the second sub-region are merged to obtain a corresponding merged attention output; and the merged attention output is decoded according to the task requirements.

[0060] Optionally, the device further comprises:

[0061] An encoding module, used for obtaining a bounding box of the object instance, and encoding the bounding box and the category of the object instance to generate a hidden vector;

[0062] An integration module, used to integrate the hidden vector and the initial image feature through a gating mechanism and a self-attention mechanism to construct a merged image feature;

[0063] The synthetic data generation module is specifically used to decode the self-attention output of the first sub-region and the merged image features according to the task requirements.

[0064] Optionally, the encoding module is specifically used for:

[0065] The position information of the bounding box of the object instance is converted into a first representation vector in a high-dimensional feature space, and the category of the object instance is converted into a second representation vector; the first representation vector is connected with the second representation vector to obtain a combined vector; and the combined vector is mapped to a latent space using a nonlinear transformation of a neural network to generate the latent vector.

[0066] Optionally, the integration module is specifically used for:

[0067] Performing feature connection between the hidden vector and the initial image feature to obtain a first feature representation;

[0068] Performing a linear transformation on the first feature representation according to a preset third weight matrix, and applying the self-attention mechanism to obtain a self-attention output of the first feature representation;

[0069] A visual tag selection operation and a gating mechanism are applied to the self-attention output of the first feature representation to generate the merged image feature; the visual tag selection operation is used to select the tag most relevant to the task requirement from the self-attention output of the first feature representation.

[0070] Optionally, the integration module, when applying a visual marker selection operation and a gating mechanism to the self-attention output of the first feature representation to generate the merged image feature, includes:

[0071] A visual marker selection operation is applied to the self-attention output of the first feature representation to obtain a second feature representation; a gating mechanism is applied to the second feature representation, and the corresponding gating mechanism output is added to the initial image feature to generate the merged image feature; the gating mechanism adjusts the contribution of the second feature representation to the initial image feature through a learnable gating parameter and a hyperbolic tangent function.

[0072] Optionally, when the integration module is used to perform a linear transformation on the first feature representation according to a preset third weight matrix and apply the self-attention mechanism to obtain a self-attention output of the first feature representation, it includes:

[0073] Obtaining a task-adaptive embedding; the task-adaptive embedding comprising a plurality of learnable embedded task-adaptive cues of the same dimension as the initial image features;

[0074] Performing feature concatenation on the first feature representation and the task adaptive embedding along the spatial dimension to obtain a merged feature representation;

[0075] Using a third weight matrix for generating a query vector, linearly transforming the first feature representation to generate a first query vector, and using a third weight matrix for generating a key vector and a value vector, linearly transforming the merged feature representation to generate a first key vector and a first value vector;

[0076] A self-attention mechanism is applied to the first query vector, the first key vector, and the first value vector to obtain a self-attention output of the first feature representation.

[0077] Optionally, the device further comprises:

[0078] Acquire a conditional embedding associated with the task requirement; the conditional embedding includes an embedding vector obtained by encoding specific generation condition information; and perform feature connection between the conditional embedding and the merged feature representation along the spatial dimension to obtain an updated merged feature representation.

[0079] Optionally, the instance attention output module is specifically used for:

[0080] The second image feature is linearly transformed using weight matrices for generating a query vector, a key vector, and a value vector, respectively, to generate a target query, a target key, and a target value for the object instance; the self-attention mechanism is applied to the target query, the target key, and the target value to obtain a self-attention output for the first sub-region.

[0081] Optionally, the synthetic data generating module is specifically used for:

[0082] Determine a target decoder for achieving the task requirement from a plurality of pre-set decoder architectures; input the self-attention output of the first sub-region into the target decoder to obtain target synthetic data output by the target decoder.

[0083] According to a third aspect of an embodiment of the present application, an electronic device is provided, comprising: a memory and a processor; the memory is used to store a computer program; the processor is used to execute the above-mentioned synthetic data generation method by calling the computer program.

[0084] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the above-mentioned synthetic data generating method is implemented.

[0085] The technical solution provided by the embodiments of the present application may have the following beneficial effects:

[0086] In the technical solution provided by the present application, an initial image feature is generated based on the user's demand data, and then the image feature of each object instance is separated from the initial feature, and the category information of each object instance is embedded into the image feature of the area where the object instance is located. The self-attention output of the image feature with added category information is obtained by combining linear transformation with the self-attention mechanism to generate synthetic data, so that object instances of different categories can be distinguished and their feature representation changes can be accurately controlled during the synthetic data generation process. Users can adjust the corresponding scene elements according to their needs, thereby achieving a high degree of controllability of the scene during the synthetic data generation process.

[0087] It should be understood that the above general description and the following detailed description are only exemplary and explanatory and cannot limit the present application. In addition, any embodiment in the present application does not need to achieve all the above effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0089] Figure 1 is a flowchart of a synthetic data generation method shown in an exemplary embodiment of the present application;

[0090] Figure 2 It is a flowchart of another synthetic data generation method for adding empty instance prompts shown in an exemplary embodiment of the present application;

[0091] Figure 3A It is a flowchart of another synthetic data generation method that adds gated self-attention application to instance boxes, shown in an exemplary embodiment of the present application;

[0092] Figure 3B It is a flowchart of a step of applying a gated self-attention mechanism to construct merged image features as shown in an exemplary embodiment of the present application;

[0093] Figure 3C It is a flowchart of another step of introducing task adaptive prompts and applying gated self-attention mechanism to construct merged image features as shown in an exemplary embodiment of the present application;

[0094] Figure 4 is a structural schematic diagram of a synthetic data generating device shown in an exemplary embodiment of the present application;

[0095] Figure 5 It is a hardware schematic diagram of an electronic device shown in an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0096] Synthetic data is data created in the digital world that imitates real-world data, rather than being collected or measured from the real world. It simulates the data distribution and patterns in the real world and is created by computational algorithms and simulations based on generative artificial intelligence technology. It has the same mathematical properties as the actual data on which it is based but does not contain the same information. It can mathematically or statistically reflect the properties of real-world data and can be used as a substitute for real-world data to train machine learning models, test the performance of algorithms in different scenarios, and provide large amounts of data while protecting personal privacy. The generation of synthetic data can be based on statistical models, deep learning techniques (such as generative adversarial networks (GANs), or other computational methods to ensure that the generated data has similar properties to real data.

[0097] At present, the development of autonomous driving and embodied intelligence relies on large computing power and big data, which is the data set required for training and testing. At present, this data set is usually obtained in the following two ways: one is real data collected through real physical environment, and the other is simulation data generated by virtual environment constructed by data twin simulation technology. Real data can reflect the actual traffic environment, but because effective scenarios such as long-tail scenarios or challenging scenarios are difficult to capture, the amount of available data in real data is often insufficient relative to the amount of data required for development, and the cost of collecting such data is high and often restricted by actual conditions, making it difficult to be widely popularized. In contrast, the simulation data generated by data twin simulation technology has effectively alleviated the problem of insufficient real data to a certain extent. Digital twin simulation data can simulate specific scenarios, but it has poor fidelity and logical coherence and cannot fully reflect all the details and dynamics in the real world.

[0098] The current simulation data generation process makes it difficult to independently control specific static or dynamic elements in a refined manner while maintaining the overall semantics of the scene. For example, for static environmental factors such as road types, weather conditions, and time changes, as well as dynamic traffic elements such as vehicle types and motion trajectories, existing simulation technologies often find it difficult to achieve precise adjustment and diversified combinations of these elements while maintaining scene coherence and consistency. This lack of controllability in data generation scenarios limits the effectiveness and efficiency of synthetic data in enhancing data diversity and quality, and also affects its effectiveness and efficiency in autonomous driving and embodied intelligence training.

[0099] In view of this, the present application provides a method for generating synthetic data, which can be implemented on a pre-trained neural network for generating synthetic data, such as a world model, or other models that can simulate the distribution of real-world data to generate synthetic data, and can be executed by any electronic device that supports running software or neural networks. This method is suitable for generating training data required for development and research in the field of autonomous driving or other embodied intelligence fields such as intelligent robots, and adopts a new image feature combined with an attention mechanism processing process to improve the controllability of the dynamic and static elements of the scene during the generation of synthetic data, as well as the consistency and coherence of the scene.

[0100] See also Figure 1 As shown in the flowchart of the method steps, the synthetic data generation method provided by the present application may at least include the following steps:

[0101] S101, obtaining initial image features corresponding to user demand data;

[0102] The user demand data is used to guide and control the generation process of the synthetic data to generate synthetic data with specific attributes and conditions that meet the user's needs. The output of the synthetic data generation method is the target synthetic data, that is, data that imitates the characteristics and distribution of real-world data based on the user demand data. Therefore, the user demand data may include but is not limited to real data collected from the real world, existing simulation data, control signals, basic scene layouts, synthetic data templates, text descriptions, and random noise.

[0103] Among them, the real data or simulated data can be used as a reference or starting point for generating synthetic data, and new synthetic data can be constructed by reusing, generalizing or generating new scenes for the real data or simulated data; the control signal is used to provide the conditions required to generate a specific scene, for example, it may include user-defined text descriptions, parameter settings (such as weather, time, road type, scene layout), and specific dynamic elements (such as traffic participant type, motion trajectory) to ensure that the output data meets user needs; the basic scene layout provides a predefined scene structure, which may include a basic layout of static elements (such as roads, buildings) and dynamic elements (such as vehicles, pedestrians) to add or modify details on the scene framework, for example, providing a blank image containing a two-dimensional detection frame of a vehicle; the random noise is used to introduce changes to ensure that the generated scenes are diverse and avoid data duplication.

[0104] The user demand data is determined according to the task requirements when actually generating synthetic data. For example, for the existing training data reuse scenario, the user demand data may include the original training data that needs to be reused; for the existing training data generalization scenario, the user demand data may include the basic training data and generalization direction of the generalization process; for the new scenario of using existing training data to generate future predictions, the demand data may include the historical time series data required for prediction.

[0105] The user demand data may include data of the type of text, image, map input, etc., and image data may include scene images, and / or semantic segmentation maps, scene segmentation maps. The initial image features are used to accurately reflect key information such as the content and layout of the image scene, the location and category of object instances, and spatial relationships in the user demand data. The key information may also include detailed information such as the depth, texture, and lighting of the scene, depending on the content of the user demand data.

[0106] In the case where the user demand data is input as a text description, the initial image features can be obtained by converting the text description into a visualized scene image. Based on the results of semantic segmentation and object instance recognition of the text description, an accurate segmentation mask can be constructed in a blank or predefined scene. The segmentation mask corresponds to the object instance in the text description and can describe the shape and boundary of the object instance to construct a scene image corresponding to the text description and convert the abstract text information into specific image features. This process can be implemented through a SAM (Segment Anything Model) model or other models or algorithms that can realize the conversion of text description to image features.

[0107] When the user demand data is an image type input, the corresponding initial image features can be obtained through the transformer-based architecture Vision Transformer, or other image feature extraction methods such as a mixed use of traditional convolution and Transformer architectures.

[0108] S102, obtaining a first image feature of a first sub-region where each object instance in the initial image feature is located, and adding a learnable category hint of the object instance to the first image feature to obtain a second image feature;

[0109] The object instance refers to the physical object described in the user demand data, such as a pedestrian, a car or a traffic light, and each object instance is a separate instance of the same type of object, which is identified by a target or object detection algorithm. The first sub-region refers to the image region associated with each object instance, which can be determined based on the bounding box of the object instance, such as directly taking the region where the bounding box is located as the first sub-region where the object instance is located.

[0110] The first image feature of the first sub-region refers to a feature vector used to describe the region where the object instance is located. Based on the input being the initial image feature, the first image feature of the first sub-region where the object instance is located is extracted from the initial image feature, that is, the local features of a specific object instance are separated from the global features of the image, and the first image feature of the first sub-region where each object instance is located can be cropped from the global initial image feature according to the segmentation mask of the object instance in the initial image feature. This process can be implemented by matrix multiplication, where the segmentation mask of the object instance is multiplied by the global feature map, and only the feature information of the corresponding object is retained, while the background features are set to zero or ignored, thereby realizing the individual extraction of the features of each object instance to form the first image feature.

[0111] Among them, regarding the segmentation mask of the object instance in the initial image feature, when the user demand data is a text description input, the segmentation mask is constructed by converting the text description into a visualized scene image; when the user demand data is an image type input, the segmentation mask can be obtained by performing object detection and segmentation on the input image, that is, a target detection model (such as a fast region convolutional neural network, a neural network YOLO) can be used to perform object detection on the input image to identify each object instance and its position in the image, and then a segmentation mask for each object instance is generated through a semantic segmentation or instance segmentation model such as a mask region convolutional neural network, a deep learning semantic segmentation or a U-shaped network.

[0112] For the case where the initial image feature does not include the segmentation mask of the object instance, based on the fact that the initial image feature includes abstract information such as the shape, category and position of the object instance described by the user demand data, a candidate region containing the object instance can be located by utilizing a deep learning-based region proposal network or a similar mechanism, combined with abstract information such as the shape, category and position of the object instance, and the first image feature corresponding to the candidate region in the initial image feature can be extracted.

[0113] The learnable category hint refers to information that is dynamically adjusted and optimized during the model training process. The category hint is used to represent the feature representation of the category of the object instance, and is used to guide the model to better understand and distinguish object instances of different categories. The category hint usually exists in the form of a vector, each vector represents a category and contains information or attributes unique to the category. The category hint is continuously iterated and updated during the training process to better combine with the image features, and is shared among all the attention layers of the model, ensuring a consistent understanding of the object instances at different levels.

[0114] Combining the category hint of each object instance with its corresponding first image feature can be achieved through a simple splicing operation. The first image feature of the first sub-region where each object instance is located is added with the category hint of the corresponding object instance, thereby obtaining the second image feature of the first sub-region. The second image feature not only contains the image information of the object instance, but also incorporates the semantic information at the category level of the object instance, so that the model can take into account both the image features and category information of the object instance, which is convenient for precise control of each object instance during the synthetic data generation process.

[0115] Suppose there is an image containing a cat and a dog, identify the location and category of the cat and dog, and assign corresponding instance hints e to the cat and dog. cat and e dog , the image feature r in the area where the cat is located cat Add e cat , and similarly for the image feature r of the area where the dog is located dog Will add e dog , generating the corresponding second image features.

[0116] S103, performing a linear transformation on the second image feature according to a preconfigured first weight matrix, and applying a self-attention mechanism to obtain a self-attention output of the first sub-region;

[0117] The self-attention mechanism is a key architectural component that allows the model to focus on different parts of the input sequence, thereby capturing more complex dependencies when processing long sequences of data.

[0118] In the self-attention mechanism, the preconfigured first weight matrix includes W Q , W K and W VA matrix is ​​used to convert the input feature vector into a query vector, a key vector and a value vector. The weight matrix is ​​learned by the back propagation algorithm during the model training process. The linear transformation is a process of operating the input vector by matrix multiplication. Before applying the self-attention mechanism, the second image feature is linearly transformed by multiplication with the preconfigured first weight matrix to obtain the query vector, the key vector and the value vector.

[0119] Based on this, this step can be implemented in the following manner: using the first weight matrix that generates the query vector, key vector and value vector respectively, linearly transform the second image feature to generate the target query, target key and target value of the object instance; applying the self-attention mechanism to the target query, target key and target value to obtain the self-attention output of the first sub-area.

[0120] Regarding the application of the self-attention mechanism to obtain the self-attention output, after obtaining the target query, target key and target value, the self-attention mechanism calculates the dot product of the target query and all target keys, and then normalizes it through the softmax function to obtain the attention weight, which represents the degree of "attention" of each sub-region to the first sub-region. The attention weight is then weighted and summed with all target values ​​to obtain the self-attention output of the first sub-region, which contains the feature representation of the first sub-region.

[0121] Regarding the linear transformation of the second image feature according to the preconfigured first weight matrix, it can be implemented by a preconfigured linear projection layer, which is used to convert the input data into query, key and value vectors by applying a linear transformation. For example, the linear projection layer can be expressed as Then, performing a linear transformation on the second image feature to obtain the target query, target key and target value can be expressed as:

[0122]

[0123] Among them, r (i) represents the first image feature of the first subregion of the i-th object instance of category k, r (i) =a (i) +e k represents the second image feature after adding category hints, represents a linear projection layer shared across object instances, represents the target query, represents the target key, Represents the target value. Based on this, the self-attention output of the first sub-region obtained by applying the self-attention mechanism can be expressed as:

[0124]

[0125] S104: Decode the self-attention output of the first sub-region according to task requirements to generate target synthetic data.

[0126] Task requirements refer to the specific goals or processing direction requirements that need to be achieved in decoding and generating synthetic data, which may include but are not limited to data reuse, data generalization processing, and generating new data scenarios. Data reuse refers to changing a certain condition or parameter in the data set so that the consistency of the original data set scene is maintained while adapting to new scenarios or task requirements. For example, scene images under specific lighting conditions and backgrounds cannot be applied, and can be adjusted to scene images adapted to the new environment through data reuse. Data generalization processing refers to generating new data that maintains the same scene consistency as the original training data set by changing different dynamic or static elements of the original data set. Generating new data scenarios refers to generating new scenes or data that have never been seen before based on existing knowledge and data. The new data is innovative and maintains consistency and coherence with the original data.

[0127] Decoding refers to the process of converting the self-attention output of the first sub-region into the final output, which can be achieved by a decoder. The decoder is used to recover meaningful information from the features extracted by the encoder and convert it into a form suitable for specific task requirements. Therefore, the architecture of the decoder can be designed separately according to the task requirements. For example, the decoder under the image generation task requirements may include a series of upsampling layers to gradually reconstruct the image. The decoder may include a series of layers, such as an input layer, a hidden layer, and an output layer. The input layer is used to receive the features to be decoded. The hidden layer is designed according to the complexity and requirements of the task. The multi-layer neural network is used to further process the features and gradually construct the representation of the second synthetic data. It may include a convolutional layer, a recurrent neural network layer, or a self-attention layer, etc. The output layer is used to generate target synthetic data in a format that meets the task requirements. In this embodiment, the input layer of the decoder receives the self-attention output of the first sub-region, and the output layer is used to output target synthetic data that meets the task requirements.

[0128] Based on the structural design of the decoder, this step can be achieved through the following steps: determine the target decoder for achieving the task requirements from a plurality of pre-set decoder architectures; input the self-attention output of the first sub-region to the target decoder to obtain the target synthetic data output by the target decoder. That is, corresponding decoders are set in advance for different task requirements, so that after obtaining the self-attention output of the first sub-region, the corresponding decoder can be called to process the self-attention output of the first sub-region to generate the target synthetic data that meets the task requirements.

[0129] In the process of inputting the self-attention output of the first sub-region into the target decoder, the self-attention outputs of each first sub-region can be aggregated in advance by weighted averaging or other fusion features, and the fused features can be integrated into the target decoder for processing.

[0130] For the above decoding process, in order to enable the decoder to change the features according to the self-attention output of the first sub-region, the decoder may include a conditional generation module to understand and respond to different task requirements, and select the features that need to be changed in the self-attention output of the first sub-region and make corresponding adjustments according to the pre-learned processing method or additional task condition input. For example, the task requirement is data generalization processing. In the absence of additional task condition input, assuming that the pre-learned data generalization processing rules include but are not limited to changing weather conditions, the color, texture, position or size of the object instance, the decoder can analyze the self-attention output of the first sub-region, identify the part corresponding to the adjustable feature, and selectively apply the rules according to the pre-defined processing method to modify the features in the self-attention output. For example, change the weather in the scene from sunny to rainy, find and adjust the weather-related features in the output; change the color of an object instance from red to blue, and update the features of the color information of the object instance in the output to generate diversified synthetic data according to the task requirements to meet different application scenarios and user needs.

[0131] In the disclosed embodiment, the initial image features are generated based on the user's demand data, and then the image features of each object instance are separated from the initial feature, and the category information of each object instance is embedded into the image features of the area where the object instance is located. The self-attention output of the image features with added category information is obtained by combining linear transformation with the self-attention mechanism to generate synthetic data, so that object instances of different categories can be distinguished during the generation of synthetic data and their feature representation changes can be accurately controlled. The user can adjust the corresponding scene elements according to the needs, thereby achieving a high degree of controllability of the scene during the generation of synthetic data. At the same time, the self-attention mechanism and linear transformation effectively extract the relationship between the category, attribute and object instance of the object instance, so that the generated synthetic data and the user's demand data maintain the logical rationality and layout consistency in the scene, providing rich, diverse and controllable synthetic data resources for application scenarios such as machine learning, simulation, training and testing.

[0132] In some embodiments, based on the goal of improving the consistency and controllability of scene generation, in order to more accurately process background areas that do not contain any object instances, it is ensured that the synthetic data generation process can reasonably generate and understand background information and avoid confusion with object instances in the foreground area, such as Figure 2 As shown, in Figure 1On the basis of the described embodiments, a processing process for a sub-image region (i.e., a background region) that does not contain any object instance may also be included. The processing process may include the following steps:

[0133] S201, obtaining a third image feature of a second sub-region that does not contain any object instance in the initial image feature, and adding a learnable empty instance hint to the third image feature to obtain a fourth image feature;

[0134] Corresponding to the first sub-region where the object instance described in the aforementioned step S102 is located, the second sub-region refers to a region that does not contain any object instance, which may be a background region on an image containing an object instance, or may be an image region where no object instance exists in the entire image.

[0135] After the third image feature of the second sub-region is separated from the initial image feature, in order to improve the attention layer's ability to understand and generate background information, a learnable empty instance hint is added to the third image feature. The empty instance hint is a special vector that indicates that the current processing is a background region that does not contain an object instance. It will be learned and optimized during the training process to better distinguish and generate background information. The fourth image feature obtained in this way not only contains the visual information of the second sub-region, but also contains additional hint information indicating that it does not contain an object instance.

[0136] S202, performing a linear transformation on the fourth image feature according to a preconfigured second weight matrix, and applying the self-attention mechanism to obtain a self-attention output of the second sub-region;

[0137] The method of obtaining the self-attention output of the second sub-region in this step is the same as the method of obtaining the self-attention output of the first sub-region in the aforementioned step S103. The fourth image feature is linearly transformed using the second weight matrix to obtain the corresponding query vector, key vector and value vector, and then the query vector, key vector and value vector are calculated using the Attention() of the self-attention mechanism to capture the dependency between different parts in the background area and obtain a more refined and accurate background feature representation, so that a more coherent and natural background can be generated in the process of synthetic data generation.

[0138] The second weight matrix may be directly set to be the same as the first weight matrix described in the aforementioned step S103, or the second weight matrix may be set independently for a scene added to an empty instance and learned through a back propagation algorithm during model training.

[0139] Based on the above-mentioned added processing process for the sub-region that does not contain any object instance, in this embodiment, step S104 is expanded to apply the self-attention output of the second sub-region to the synthetic data generation process, that is, the above-mentioned step S104 is updated to the illustrated step S203, and the self-attention output of the first sub-region and the self-attention output of the second sub-region can be decoded according to the task requirements to ensure that the information of the object instance in the foreground area and the background area can be fully considered and used to generate the final synthetic data, thereby improving the integrity and consistency of the entire scene. That is, the self-attention output of the first sub-region containing the object instance and the self-attention output of the second sub-region that does not contain the object area will be used as input information for the decoding process at the same time, and the input information will be decoded according to the task requirements.

[0140] In order to integrate the key information of different sub-regions into a unified representation, when the first sub-region and the second sub-region are used as input information for decoding processing, the self-attention outputs of the first sub-region and the second sub-region can be merged according to the mask information of the first sub-region and the second sub-region in the initial image features to obtain the corresponding merged attention output, and then the merged attention output is used as the input information for decoding processing, and the merged attention output is decoded according to the task requirements.

[0141] The mask information of the first sub-region and the second sub-region in the initial image feature is a binary information used to distinguish the first sub-region from the second sub-region. In the case where the user demand data is an image type input, the mask information can be obtained through image segmentation technology, and the remaining area is marked as the second sub-region based on the first sub-region where the object instance is detected; in the case where the user demand data is a text description input, after constructing the segmentation mask of the object instance, other areas outside the object instance in the blank or predefined background can be marked as the second sub-region, thereby obtaining the mask information.

[0142] The merging of the attention outputs of two different sub-regions can be achieved through simple weighted superposition or more complex fusion strategies, and the merging method can be selected according to the actual task requirements. By merging the self-attention outputs of the two sub-regions, the foreground and background information can be integrated in a unified representation, and the merged attention output contains richer information, which is conducive to a more comprehensive understanding of the image content to generate target synthetic data that meets the task requirements, while maintaining scene consistency and logical rationality.

[0143] In the disclosed embodiments, while adding specific learnable category embeddings to the areas where object instances are located, empty instance hints are introduced and added to the image features of areas that do not contain object instances, so that the synthetic data generation process can distinguish between foreground areas and background areas and effectively process the background areas, thereby achieving the overall consistency of the generated synthetic data scenes and avoiding the lack or unnaturalness of background information.

[0144] In some embodiments, in order to optimize and enhance the feature representation of the instance box of the object instance (i.e., the bounding box of the object instance) while ensuring the consistency and controllability of the instance box in the process of generating synthetic data, the present embodiment Figure 1 On the basis of adding category hints to the area where the object instance is located, a step is added to optimize the features of the instance box using a gated self-attention mechanism, see Figure 3A As shown in Figure 1, the gated self-attention mechanism can be used to optimize the features of the instance box by the following steps:

[0145] S301, obtaining a bounding box of the object instance, and encoding the bounding box and the category of the object instance to generate a hidden vector;

[0146] The bounding box of the object instance is a rectangular box used to locate the position of an object instance in the image. It can be defined by four coordinate values ​​to accurately indicate the position and size of the object. Obtaining the bounding box of the object instance is essentially obtaining the position information of the bounding box. Each object instance is classified into a specific category, such as "dog", "cat", "car", etc. The category information is used to distinguish different object instances.

[0147] Hidden vectors are usually used in computer vision and deep learning to store additional information about data, which is information that needs to be specially encoded during the reasoning process. In this embodiment, the hidden vector is used to store additional information of each object instance, such as the category of the instance and the location information of the bounding box or other related features, to capture the structure and relationship of the instance boxes of different object instances, so as to achieve fine control over specific instances when generating synthetic data.

[0148] Encoding the bounding box and the category of the object instance to generate a hidden vector can be achieved in the following manner: converting the position information of the bounding box of the object instance into a first representation vector in a high-dimensional feature space, and converting the category of the object instance into a second representation vector, wherein the second representation vector contains semantic information of the category of the object instance; connecting the first representation vector and the second representation vector to obtain a combined vector; and using a nonlinear transformation of a neural network to map the combined vector to a hidden space to generate the hidden vector.

[0149] Among them, the position information can be converted into a first representation vector by Fourier embedding or position encoding, machine learning, etc., and the category of the instance object can be converted to generate the second representation vector by text embedding representation or pre-trained models such as CLIP (Contrastive Language-Image Pre-training). After obtaining the first representation vector and the second representation vector, the two vector representations are concatenated using a [,] connection operation to obtain a combined vector. Regarding the use of a nonlinear transformation of a neural network to convert the combined vector into a hidden vector, a multi-layer perceptron can be used to map the combined vector to generate a fixed-length vector representation as the hidden vector.

[0150] S302, integrating the hidden vector and the initial image feature through a gating mechanism and a self-attention mechanism to construct a merged image feature;

[0151] The gating mechanism is a way to control the flow of information in a neural network by introducing one or more "gates" to control the contribution of information. The "gate" is automatically learned by the network and dynamically adjusts its open or closed state based on the input data. In sequence processing and attention layers, the gating mechanism is widely used to improve the performance and learning ability of the model.

[0152] In this step, the hidden vector and the initial image features obtained based on user demand data are taken as input, and the self-attention mechanism is used to capture the dependency between the instance boxes of different object instances in the image and the various regions on the initial image features. The contribution of the processing results of the self-attention mechanism to the initial image features is adjusted through the gating mechanism, thereby constructing a more comprehensive and context-rich merged image feature. The merged image feature enhances the feature representation capability of the instance box and the internal image and reduces the interference of background noise.

[0153] Based on the introduction of the above-mentioned gated self-attention mechanism, in this embodiment, step S104 is expanded to apply the merged image features constructed by the gated self-attention mechanism using the hidden vector to the synthetic data generation process, that is, the above-mentioned step S104 is updated to the illustrated step S303, and the self-attention output of the first sub-region and the merged image features can be decoded according to the task requirements.

[0154] When the self-attention output of the first sub-region and the merged image features are used as input information for decoding processing, the self-attention output can be fused with the merged image features by simple splicing, weighted sum or using a more complex fusion network to combine the information in the two feature spaces so that subsequent decoding processing can make full use of this information.

[0155] In the disclosed embodiments, the bounding box of the object instance is encoded together with the category information into a hidden vector, so that different object instances can be identified and distinguished when processing different object instances. At the same time, the self-attention mechanism is used to focus on the overall layout and contextual information in the initial image features, thereby enhancing the richness of the feature representation of each object instance. The gating mechanism is combined to dynamically adjust the attention to different object instance features, thereby achieving precise control of the instance frame, and maintaining consistency between the categories of the object instances and the layout and position of the instance frames in the subsequent feature integration and decoding process.

[0156] In some embodiments, for the above step S302, the hidden vector and the initial image feature are integrated through the gating mechanism and the self-attention mechanism to construct the merged image feature, see Figure 3B As shown, this step can be implemented as follows:

[0157] S312, performing feature connection on the hidden vector and the initial image feature to obtain a first feature representation;

[0158] In this embodiment, the feature concatenation of the latent vector and the initial image feature refers to concatenating the two sets of vectors in rows or columns, and the obtained first feature representation includes all the information of the latent vector and the initial image feature.

[0159] In the case of an instance frame containing multiple object instances, each instance frame has a corresponding hidden vector. In this step, the hidden vectors of all instance frames are first integrated together, and then the integrated result is spliced ​​with the initial image features. For example, the hidden vectors corresponding to each instance frame can be integrated into one hidden vector through the segmentation mask of each object instance, or multiple hidden vectors can be integrated with the initial image features respectively, and the optimized feature representation is generated through the self-attention mechanism of step S312, and then the obtained multiple optimized feature representations are further integrated according to the segmentation mask of the object instance to ensure that the feature optimization results of all instance frames in the scene can be effectively integrated together to ensure the consistency and integrity of the scene.

[0160] S322, performing a linear transformation on the first feature representation according to a preset third weight matrix, and applying the self-attention mechanism to obtain a self-attention output of the first feature representation;

[0161] The third weight matrix used for linear transformation in this embodiment can be the same as the weight matrix used for linear transformation in the aforementioned step S103, and the same linear transformation can be used. Alternatively, the second weight matrix can also be independently set for constructing merged image features and learned through the back propagation algorithm during model training. After receiving the first feature representation, the first feature representation is linearly transformed using the third weight matrix to generate the corresponding query vector, key vector and value vector. The self-attention output of the first representation can be obtained by applying the self-attention calculation Attention() to the query vector, key vector and value vector.

[0162] S332, applying a visual tag selection operation and a gating mechanism to the self-attention output represented by the first feature to generate the merged image feature; the visual tag selection operation is used to select the tag that is most relevant to the task requirement from the self-attention output represented by the first feature.

[0163] The visual marker selection operation is a technique used in the self-attention mechanism to filter out the visual markers that are most relevant to the current task requirements from the output of the self-attention mechanism. In visual tasks, the self-attention mechanism can process different parts of the input image and assign different weights to capture the global dependencies of the image. The visual marker selection operation can help the model focus on the most critical visual features for completing specific tasks through this mechanism. Based on the fact that each marker token output by the self-attention mechanism represents the key area or object in the input image, this operation evaluates each marker output by the self-attention mechanism and selects the most useful marker subset according to the goal of the task requirements, thereby improving the efficiency and accuracy of processing visual information.

[0164] The gating mechanism acts on the output result of the visual marker selection operation, and generates the merged image feature by controlling the contribution of the output result to the initial image feature. The merged image feature contains the optimized feature information of all instance boxes in the scene, rather than multiple outputs generated independently for each instance box, thereby achieving fine control and optimization of multiple instance boxes in the scene while maintaining the integrity and consistency of the scene.

[0165] In some embodiments, the gating mechanism can adjust the contribution of the output result of the visual mark selection operation to the initial image feature through learnable gating parameters and hyperbolic tangent functions, that is, the above-mentioned application of the visual mark selection operation and the gating mechanism to the self-attention output of the first feature representation can be implemented as follows: applying the visual mark selection operation to the self-attention output of the first feature representation to obtain a second feature representation; applying the gating mechanism to the second feature representation, and adding the corresponding gating mechanism output to the initial image feature to generate the merged image feature.

[0166] Based on the above hidden vector and gating mechanism implementation, the process of constructing the merged image features according to the hidden vector and the initial image features can be expressed by the formula:

[0167]

[0168] Wherein, x on the right side of the formula represents the initial image feature; η is a learnable parameter used to adjust the degree of fusion between the initial image feature x and the output result of TS(); tanh() represents the hyperbolic tangent function; TS() represents the visual marker selection operation; () represents the self-attention mechanism; [,] represents the feature vector connection operation; H b Represents a hidden vector. When using fourier() embedding, text embedding representation, and multi-layer perceptron to obtain a hidden vector, the process of obtaining the hidden vector can be expressed as:

[0169]

[0170] Among them, MLP() represents the mapping process of the multi-layer perceptron, and the bounding box Can be a 2D box or a 3D box, bounding box type Where N is the number of instance boxes of the object instance.

[0171] To better understand the above hidden vectors and merged image features, we will take the input scenario of "the image in front of the vehicle, including a traffic light and a pedestrian" as an example. b It refers to the hidden vector containing the position information and category information of the bounding box of traffic lights and pedestrians; the initial image feature X on the right side of the formula represents the image feature encoding of the input scene, which contains the visual information of the background area and all object instances; TS() is used to select the visual mark that is most relevant to the current task (such as optimizing the features of pedestrians or traffic lights) from the output of the self-attention mechanism. For example, if the current task is to optimize the position of pedestrians in the scene, then TS() will be more inclined to select marks associated with pedestrian features; X, which is finally output by the gated self-attention mechanism, is a scene graph feature encoding for the entire scene that contains the optimization results of traffic lights and pedestrian features, which more accurately expresses the relative position of pedestrians in the scene and the state of traffic lights, ensuring that the position, category and fusion of all instance boxes in the scene with the background are better optimized, thereby ensuring the controllability and accuracy of details of the generated scene.

[0172] In the disclosed embodiment, by feature-connecting the hidden vector with the initial image features, the high-level semantic information contained in the hidden vector can be directly integrated into the image feature generation process, and the feature tags most relevant to the task requirements are extracted from the self-attention output through the visual tag selection operation. The selective attention mechanism enables precise control of which features should be emphasized or ignored when generating specific scenes or object instances, thereby improving the controllability of scene generation. At the same time, the controllability of the instance box is further enhanced by combining the gating mechanism, and the importance of different features can be dynamically adjusted according to task requirements, and irrelevant noise or redundant information can be filtered out, thereby enhancing the high controllability of the scene in the process of synthetic data generation.

[0173] In some embodiments, based on the diversity of user needs and the requirement for precise control of scene elements, in order to further improve the flexibility and adaptability of the data generation model, in the process of applying the gated self-attention mechanism to construct the merged image features, when applying self-attention calculation to the feature concatenation result of the hidden vector and the initial image features, that is, the first feature representation, the task-adaptive embedding is introduced to jointly determine the query vector, key vector and value vector required for the self-attention mechanism calculation.

[0174] Based on this, Figure 3C As shown, the linear transformation of the first feature representation according to the preset third weight matrix described in the aforementioned step S322, and the application of the self-attention mechanism to obtain the self-attention output of the first feature representation, can also be achieved in the following manner:

[0175] S3221, obtaining a task-adaptive embedding; the task-adaptive embedding includes a plurality of learnable embedded task-adaptive prompts of the same dimension as the initial image feature;

[0176] Task-adaptive embeddings include a set of embeddings that are dynamically adjusted according to the current task requirements, which are used to enhance the flexibility and adaptability of the model when handling different tasks. They contain a series of learnable parameters, namely embeddings, which can be adjusted according to different task requirements. Different features or patterns need to be focused on in different task requirements. The embedding vector can be adjusted to highlight the most useful information for a specific task. During the training process, the parameters are initialized through Gaussian distribution to avoid the problem of gradient vanishing or exploding. At the same time, the network is allowed to explore various possible weight values ​​at the beginning of training, and the task-adaptive embedding is learned during the training phase.

[0177] Task adaptive embedding is a technique used in deep learning models to improve the flexibility and adaptability of models in handling different tasks. It introduces a set of learnable embedding vectors (or parameters) so that the model can dynamically adjust these vectors according to the specific needs of different tasks, thereby focusing more on feature information that is useful for specific tasks.

[0178] Taking the image generalization task containing pedestrians and traffic lights as an example, the generalization task can be performed by changing the position of pedestrians or the state of traffic lights. Based on the different features, sizes, positions and other information of pedestrians and traffic lights in the image, in order to improve the pertinence and accuracy of the generalization task, the image generalization process is required to adjust its focus for each task. Therefore, a set of learnable embedding vectors can be set for the generalization task of pedestrian position change and the generalization task of traffic light state change. The vectors are randomly initialized by Gaussian distribution at the beginning of training. With continuous training on the image dataset containing pedestrians and traffic lights, the embedding vectors gradually learn the most useful features for their respective tasks. For example, for the task of pedestrian position change, the embedding vectors are gradually adjusted to pay more attention to the contour, color, texture and other features of the human body in the image, while for the task of traffic light state change, the embedding vectors may pay more attention to the specific shapes (circular or arrow shapes) and colors (red, yellow, green) in the image.

[0179] S3222, performing feature connection on the first feature representation and the task adaptive embedding along the spatial dimension to obtain a merged feature representation;

[0180] In deep learning, feature concatenation refers to directly concatenating two or more feature vectors along a certain dimension to form a larger vector. In this step, the first feature representation and the task-adaptive embedding are concatenated along the spatial dimension, that is, each dimension of the task-adaptive embedding is concatenated with the corresponding dimension of the first feature representation to create a richer merged feature representation that contains information from both feature vectors.

[0181] S3223, using a third weight matrix for generating a query vector, linearly transforming the first feature representation to generate a first query vector, and using a third weight matrix for generating a key vector and a value vector, linearly transforming the merged feature representation to generate a first key vector and a first value vector;

[0182] The self-attention calculation in this embodiment is different from the process of the aforementioned step S322 in that the input features used when generating the query vector, key vector and value vector through linear transformation change. The third weight matrix includes three sets of weights, which correspond to the generation of query vectors, key vectors and value vectors respectively. In this step, the first query vector is obtained by converting the first feature representation using the third weight matrix for generating the query vector, while the first key vector and the first value vector are obtained by linearly transforming the concatenation result of the first feature representation and the task adaptive embedding. Assuming that the task adaptive embedding is represented by g and the first feature is represented by x, the first query vector, the first key vector and the first value vector required for the self-attention calculation Attention() in this step can be expressed as:

[0183]

[0184] Wherein, when the third weight matrix is ​​the same as the first weight matrix described in the aforementioned step S103, the It is understood as the linear projection layer mentioned above, which is used to convert the input features into corresponding query vectors, key vectors, and value vectors through linear transformation.

[0185] S3224, apply a self-attention mechanism to the first query vector, the first key vector, and the first value vector to obtain a self-attention output of the first feature representation.

[0186] After obtaining the first query vector, the first key vector and the first value vector, the self-attention calculation Attention() can be directly applied to obtain the self-attention output of the above first feature representation. The self-attention output obtained after applying the self-attention mechanism contains the attention weighted information calculated according to the task adaptive embedding and the first feature representation, and pays more attention to the features most relevant to the current task.

[0187] In the embodiments of the present disclosure, the task-adaptive embedding mechanism can dynamically adjust and optimize the key parameters in the self-attention mechanism in the process of applying the gated self-attention processing instance frame according to the specific needs of the current task. By learning the task-related feature representation and embedding it into the calculation process of the self-attention mechanism, the feature information most relevant to the current task can be captured more accurately. According to the guidance of the task-adaptive embedding, the static and dynamic elements are precisely adjusted to achieve precise control of the static and dynamic elements in the scene, so as to ensure that the generated scene conforms to the laws of physics and meets the specific needs of users, thereby generating more diversified synthetic data that meets user needs.

[0188] In some embodiments, when the task adaptive embedding is introduced to adjust the adaptability of the model to different tasks, the specific conditions of the generation process can also be accurately controlled by conditional embedding to generate synthetic data that better meets user needs. Based on this, after the merged feature representation is obtained in the above step SS3222, the following steps can also be included:

[0189] Acquire a conditional embedding associated with the task requirement; the conditional embedding includes an embedding vector obtained by encoding specific generation condition information; and perform feature connection between the conditional embedding and the merged feature representation along the spatial dimension to obtain an updated merged feature representation.

[0190] Among them, conditional embedding refers to encoding specific generation condition information such as text description, layout planning, style instructions, etc. into an embedding vector, and using it as a conditional input in the generation process to guide the model to generate data that meets the generation condition information, so as to generate synthetic data according to the user's specified conditions, thereby improving the controllability and accuracy of the generated content. For example, the task requirement is to generalize the input user demand data (such as a vehicle driving scene graph). Assuming that the user needs a rich weather direction such as a generalized scene graph under hail or heavy rain scenes, the requirement for the direction of weather change can be converted into a conditional embedding.

[0191] The acquired conditional embedding and the merged feature representation are spliced ​​together along a specific dimension, and the conditional information in the conditional embedding is directly injected into the merged feature representation, forming an updated merged feature representation, which contains both the feature information of the original merged feature and the conditional information guiding the generation process, providing a richer context and more precise control for subsequent generation steps. Based on the newly added conditional embedding, the aforementioned step S3223 will obtain the first query vector, key vector and value vector based on the updated merged feature representation.

[0192] Assuming that the conditional embedding is represented as g, the task adaptive embedding is represented as g, and the first feature is represented as x, this step obtains the first query vector, the first key vector, and the first value vector required for self-attention calculation Attention() based on the updated synthetic feature representation, which can be expressed as:

[0193]

[0194] In order to flexibly adapt to the fast conversion between unconditional embedding and conditional embedding, the conditional embedding c value can be set to meet the The conditional synthetic data generation is converted into unconditional synthetic data generation, which only introduces self-attention calculation in the case of task-adaptive embedding.

[0195] In the embodiments of the present disclosure, by introducing conditional embedding to encode specific generation condition information, the user can accurately control the synthetic data generation process, such as the theme, style, layout, etc. of the content. By adjusting the parameters in the conditional embedding or providing different condition information, the synthetic data generation results can be customized to meet different needs and generate more flexible and diverse synthetic data.

[0196] Corresponding to the embodiment of the aforementioned synthetic data generation method, see Figure 4 As shown, the present application also provides an embodiment of a synthetic data generating device, the device comprising:

[0197] The initial image feature acquisition module 401 is used to acquire the initial image features corresponding to the user demand data;

[0198] An instance category hint adding module 402 is used to obtain a first image feature of a first sub-region where each object instance is located in the initial image feature, and add a learnable category hint of the object instance to the first image feature to obtain a second image feature;

[0199] an instance attention output module 403, configured to perform a linear transformation on the second image feature according to a preconfigured first weight matrix, and apply a self-attention mechanism to obtain a self-attention output of the first sub-region;

[0200] The synthetic data generation module 404 is used to decode the self-attention output of the first sub-region according to task requirements to generate target synthetic data.

[0201] In some embodiments, the apparatus further comprises:

[0202] Acquire a third image feature of a second sub-region that does not contain any object instance in the initial image feature, and add a learnable empty instance hint to the third image feature to obtain a fourth image feature;

[0203] Performing a linear transformation on the fourth image feature according to a preconfigured second weight matrix, and applying the self-attention mechanism to obtain a self-attention output of the second sub-region;

[0204] The synthetic data generation module is specifically used to decode the self-attention outputs of the first sub-region and the second sub-region according to the task requirements.

[0205] In some embodiments, when the synthetic data generation module is used to decode the self-attention outputs of the first sub-region and the second sub-region according to the task requirements, it includes:

[0206] According to the mask information of the first sub-region and the second sub-region in the initial image features, the self-attention outputs of the first sub-region and the second sub-region are merged to obtain a corresponding merged attention output; and the merged attention output is decoded according to the task requirements.

[0207] In some embodiments, the apparatus further comprises:

[0208] An encoding module, used for obtaining a bounding box of the object instance, and encoding the bounding box and the category of the object instance to generate a hidden vector;

[0209] An integration module, used to integrate the hidden vector and the initial image feature through a gating mechanism and a self-attention mechanism to construct a merged image feature;

[0210] The synthetic data generation module is specifically used to decode the self-attention output of the first sub-region and the merged image features according to the task requirements.

[0211] In some embodiments, the encoding module is specifically used to: convert the position information of the bounding box of the object instance into a first representation vector in a high-dimensional feature space, and convert the category of the object instance into a second representation vector; connect the first representation vector with the second representation vector to obtain a combined vector; and use the nonlinear transformation of a neural network to map the combined vector to a latent space to generate the latent vector.

[0212] In some embodiments, the integration module is specifically used to:

[0213] Performing feature connection between the hidden vector and the initial image feature to obtain a first feature representation;

[0214] Performing a linear transformation on the first feature representation according to a preset third weight matrix, and applying the self-attention mechanism to obtain a self-attention output of the first feature representation;

[0215] A visual tag selection operation and a gating mechanism are applied to the self-attention output of the first feature representation to generate the merged image feature; the visual tag selection operation is used to select the tag most relevant to the task requirement from the self-attention output of the first feature representation.

[0216] In some embodiments, the integration module, when applying a visual marker selection operation and a gating mechanism to the self-attention output of the first feature representation to generate the merged image feature, includes:

[0217] applying a visual marker selection operation to the self-attention output of the first feature representation to obtain a second feature representation;

[0218] A gating mechanism is applied to the second feature representation, and the corresponding gating mechanism output is added to the initial image feature to generate the merged image feature; the gating mechanism adjusts the contribution of the second feature representation to the initial image feature through a learnable gating parameter and a hyperbolic tangent function.

[0219] In some embodiments, the integration module, when used to perform a linear transformation on the first feature representation according to a preset third weight matrix and apply the self-attention mechanism to obtain a self-attention output of the first feature representation, includes:

[0220] Obtaining a task-adaptive embedding; the task-adaptive embedding comprising a plurality of learnable embedded task-adaptive cues of the same dimension as the initial image features;

[0221] Performing feature concatenation on the first feature representation and the task adaptive embedding along the spatial dimension to obtain a merged feature representation;

[0222] Using a third weight matrix for generating a query vector, linearly transforming the first feature representation to generate a first query vector, and using a third weight matrix for generating a key vector and a value vector, linearly transforming the merged feature representation to generate a first key vector and a first value vector;

[0223] A self-attention mechanism is applied to the first query vector, the first key vector, and the first value vector to obtain a self-attention output of the first feature representation.

[0224] In some embodiments, the device also includes: obtaining a conditional embedding associated with the task requirement; the conditional embedding includes an embedding vector obtained by encoding specific generation condition information; and feature-connecting the conditional embedding and the merged feature representation along a spatial dimension to obtain an updated merged feature representation.

[0225] In some embodiments, the example attention output module is specifically used to:

[0226] Performing a linear transformation on the second image feature using weight matrices for generating a query vector, a key vector, and a value vector, respectively, to generate a target query, a target key, and a target value for the object instance;

[0227] Apply the self-attention mechanism to the target query, target key, and target value to obtain a self-attention output of the first sub-region.

[0228] In some embodiments, the synthetic data generation module is specifically used to:

[0229] Determine a target decoder for achieving the task requirement from a plurality of pre-set decoder architectures;

[0230] The self-attention output of the first sub-region is input into the target decoder to obtain the target synthetic data output by the target decoder.

[0231] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0232] The present application also provides an electronic device. The structural diagram of the electronic device is as follows: Figure 5 As shown, the electronic device 500 includes at least one processor 501, a memory 502 and a bus 503, and the at least one processor 501 is electrically connected to the memory 502; the memory 502 is configured to store at least one computer-executable instruction, and the processor 501 is configured to execute the at least one computer-executable instruction, thereby performing the steps of any synthetic data generation method provided in any embodiment or any optional implementation manner in the present application.

[0233] Furthermore, the processor 501 may be a Field-Programmable Gate Array (FPGA) or other devices with logic processing capabilities, such as a Microcontroller Unit (MCU) or a Central Process Unit (CPU).

[0234] An embodiment of the present application also provides another readable storage medium storing a computer program, which is used to implement the steps of any synthetic data generation method provided in any embodiment or any optional implementation manner in the present application when executed by a processor.

[0235] The readable storage medium provided in the embodiments of the present application includes, but is not limited to, any type of disk (including floppy disk, hard disk, optical disk, CD-ROM, and magneto-optical disk), ROM (Read-Only Memory), RAM (Random Access Memory), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory, magnetic card or optical card. That is, the readable storage medium includes any medium that can store or transmit information in a readable form by a device (e.g., a computer).

[0236] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A method for generating synthetic data, characterized in that: The method comprises: Obtaining initial image features corresponding to user demand data; Acquire a first image feature of a first subregion where each object instance in the initial image feature is located, and add a learnable category hint of the object instance to the first image feature to obtain a second image feature; Performing a linear transformation on the second image feature according to a preconfigured first weight matrix, and applying a self-attention mechanism to obtain a self-attention output of the first sub-region; The self-attention output of the first sub-region is decoded according to task requirements to generate target synthetic data.

2. The method according to claim 1, characterized in that The method further comprises: Acquire a third image feature of a second sub-region that does not contain any object instance in the initial image feature, and add a learnable empty instance hint to the third image feature to obtain a fourth image feature; Performing a linear transformation on the fourth image feature according to a preconfigured second weight matrix, and applying the self-attention mechanism to obtain a self-attention output of the second sub-region; The decoding process of the self-attention output according to the task requirements includes: According to the task requirements, the self-attention outputs of the first sub-region and the second sub-region are decoded.

3. The method according to claim 2, characterized in that The decoding process of the self-attention outputs of the first sub-region and the second sub-region according to the task requirement includes: Merging the self-attention outputs of the first sub-region and the second sub-region according to the mask information of the first sub-region and the second sub-region in the initial image feature to obtain a corresponding merged attention output; The combined attention output is decoded according to the task requirements.

4. The method according to claim 1, characterized in that: The method further comprises: Obtaining a bounding box of the object instance, and encoding the bounding box and the category of the object instance to generate a hidden vector; Integrating the hidden vector with the initial image feature through a gating mechanism and a self-attention mechanism to construct a merged image feature; The decoding process of the self-attention output of the first sub-region according to the task requirement includes: According to the task requirements, the self-attention output of the first sub-region and the merged image features are decoded.

5. The method according to claim 4, characterized in that The encoding process of the bounding box and the category of the object instance to generate a hidden vector includes: Converting the position information of the bounding box of the object instance into a first representation vector in a high-dimensional feature space, and converting the category of the object instance into a second representation vector; Connecting the first representation vector and the second representation vector to obtain a combined vector; The combination vector is mapped to a latent space using a nonlinear transformation of a neural network to generate the latent vector.

6. The method according to claim 4, characterized in that The step of integrating the hidden vector with the initial image feature through a gating mechanism and a self-attention mechanism to construct a merged image feature includes: Performing feature connection between the hidden vector and the initial image feature to obtain a first feature representation; Performing a linear transformation on the first feature representation according to a preset third weight matrix, and applying the self-attention mechanism to obtain a self-attention output of the first feature representation; A visual tag selection operation and a gating mechanism are applied to the self-attention output of the first feature representation to generate the merged image feature; the visual tag selection operation is used to select the tag most relevant to the task requirement from the self-attention output of the first feature representation.

7. The method according to claim 6, characterized in that The step of applying a visual marker selection operation and a gating mechanism to the self-attention output of the first feature representation to generate the merged image feature comprises: applying a visual marker selection operation to the self-attention output of the first feature representation to obtain a second feature representation; A gating mechanism is applied to the second feature representation, and the corresponding gating mechanism output is added to the initial image feature to generate the merged image feature; the gating mechanism adjusts the contribution of the second feature representation to the initial image feature through a learnable gating parameter and a hyperbolic tangent function.

8. The method according to claim 6, characterized in that The performing a linear transformation on the first feature representation according to a preset third weight matrix and applying the self-attention mechanism to obtain a self-attention output of the first feature representation includes: Obtaining a task-adaptive embedding; the task-adaptive embedding comprising a plurality of learnable embedded task-adaptive cues of the same dimension as the initial image features; Performing feature concatenation on the first feature representation and the task adaptive embedding along the spatial dimension to obtain a merged feature representation; Using a third weight matrix for generating a query vector, linearly transforming the first feature representation to generate a first query vector, and using a third weight matrix for generating a key vector and a value vector, linearly transforming the merged feature representation to generate a first key vector and a first value vector; A self-attention mechanism is applied to the first query vector, the first key vector, and the first value vector to obtain a self-attention output of the first feature representation.

9. The method according to claim 8, characterized in that The method further comprises: Obtaining a conditional embedding associated with the task requirement; the conditional embedding includes an embedding vector obtained by encoding specific generation condition information; The conditional embedding and the merged feature representation are feature-concatenated along a spatial dimension to obtain an updated merged feature representation.

10. The method according to claim 1, characterized in that The performing a linear transformation on the second image feature according to a preconfigured first weight matrix and applying a self-attention mechanism to obtain a self-attention output of the first sub-region includes: performing a linear transformation on the second image feature using a first weight matrix that generates a query vector, a key vector, and a value vector, respectively, to generate a target query, a target key, and a target value for the object instance; Apply the self-attention mechanism to the target query, target key, and target value to obtain a self-attention output of the first sub-region.

11. The method according to claim 1, characterized in that: The decoding process of the self-attention output of the first sub-region according to the task requirements to generate target synthetic data includes: Determine a target decoder for achieving the task requirement from a plurality of pre-set decoder architectures; The self-attention output of the first sub-region is input into the target decoder to obtain the target synthetic data output by the target decoder.

12. A synthetic data generating device, characterized in that: The device comprises: An initial image feature acquisition module is used to acquire initial image features corresponding to user demand data; an instance category hint adding module, configured to obtain a first image feature of a first sub-region where each object instance is located in the initial image feature, and add a learnable category hint of the object instance to the first image feature to obtain a second image feature; an instance attention output module, configured to perform a linear transformation on the second image feature according to a preconfigured first weight matrix, and apply a self-attention mechanism to obtain a self-attention output of the first sub-region; The synthetic data generation module is used to decode the self-attention output of the first sub-area according to task requirements to generate target synthetic data.

13. An electronic device, characterized in that: include: Memory, processor; The memory is used to store computer programs; The processor is used to call the computer program to implement the method according to any one of claims 1-11.

14. A readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

Images