Data processing method, device, electronic device and medium based on scribbled prompt

Abstract

The present application proposes a data processing method, device, electronic device and medium based on scribbled prompts, wherein the method includes: obtaining scribbled prompts and data to be processed corresponding to a data processing task; generating a first feature vector based on the embedding vectors and position vectors corresponding to each mark in the scribbled prompts and the data to be processed; inputting the first feature vector into at least one attention residual model, performing feature conversion and enhancement, and obtaining a second feature vector; and determining the task processing result corresponding to the data to be processed based on the second feature vector. By processing the scribbled prompts and the data to be processed through the attention residual model, the content of the data to be processed can be used to correctly understand the intention of the scribbled prompts, and the erroneous information in the scribbled prompts will not be misled, thereby ensuring the accuracy of the task processing results.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method, device, electronic device and medium based on scribbling prompts. Background Art

[0002] Prompt-based data processing tasks, such as object detection tasks and object classification tasks, are a technology that uses natural language prompts or visual prompts to guide computer vision models to perform specific tasks. The accuracy of task processing results is related to the accuracy of the prompt information. If the prompt is sloppy, that is, the prompt is not completely accurate, it may lead to inaccurate task processing results. Therefore, how to ensure the accuracy of task processing results when the prompt is not completely accurate is a technical problem that needs to be solved urgently. Summary of the invention

[0003] The present application aims to solve one of the technical problems in the related art at least to some extent.

[0004] To this end, the first objective of the present application is to propose a data processing method based on scribbled prompts.

[0005] The second objective of the present application is to provide a data processing device based on scribbled prompts.

[0006] The third objective of the present application is to provide an electronic device.

[0007] A fourth objective of the present application is to provide a computer-readable storage medium.

[0008] A fifth object of the present application is to provide a computer program product.

[0009] To achieve the above-mentioned purpose, the first embodiment of the present application proposes a data processing method based on sloppy prompts, comprising:

[0010] Get the scribbled prompts and data to be processed corresponding to the data processing task;

[0011] Generate a first feature vector based on the scribbled hint and the embedding vectors and position vectors corresponding to each mark in the data to be processed;

[0012] Inputting the first feature vector into at least one attention residual model to perform feature conversion and enhancement to obtain a second feature vector;

[0013] Based on the second feature vector, a task processing result corresponding to the data to be processed is determined.

[0014] To achieve the above-mentioned purpose, the second embodiment of the present application proposes a data processing device based on sloppy prompt, comprising:

[0015] An acquisition module, used to acquire the scribbled prompts and the data to be processed corresponding to the data processing task;

[0016] A first processing module, configured to generate a first feature vector based on the scribbled hint and the embedding vector and the position vector corresponding to each mark in the data to be processed;

[0017] A second processing module, configured to input the first feature vector into at least one attention residual model, perform feature conversion and enhancement, and obtain a second feature vector;

[0018] A result determination module is used to determine the task processing result corresponding to the data to be processed based on the second feature vector.

[0019] To achieve the above-mentioned purpose, the third aspect of the embodiment of the present application proposes an electronic device, including: a processor, and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement a data processing method based on sloppy prompts as described in the first aspect of the embodiment of the present application.

[0020] To achieve the above-mentioned purpose, the fourth aspect of the embodiment of the present application proposes a computer-readable storage medium, in which computer execution instructions are stored. When the computer execution instructions are executed by the processor, they are used to implement a data processing method based on sloppy prompts as described in the first aspect of the embodiment of the present application.

[0021] To achieve the above-mentioned purpose, the fifth aspect of the present application proposes a computer program product, including a computer program, which, when executed by a processor, implements a data processing method based on sloppy prompts as described in the first aspect of the embodiment of the present application.

[0022] The technical solution provided by this application brings at least the following beneficial effects:

[0023] The present application obtains the scribbled prompts and the data to be processed corresponding to the data processing task; generates a first feature vector based on the embedding vectors and position vectors corresponding to each mark in the scribbled prompts and the data to be processed; inputs the first feature vector into at least one attention residual model, performs feature conversion and enhancement, and obtains a second feature vector; and determines the task processing result corresponding to the data to be processed based on the second feature vector. By processing the scribbled prompts and the data to be processed through the attention residual model, the content of the data to be processed can be used to correctly understand the intention of the scribbled prompts, and the erroneous information in the scribbled prompts will not be misled, thereby ensuring the accuracy of the task processing results.

[0024] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0026] Figure 1 A flowchart of a data processing method based on scribbled prompts provided in an embodiment of the present application;

[0027] Figure 2 A flowchart of a data processing method based on scribbled prompts provided in another embodiment of the present application;

[0028] Figure 3 A schematic diagram of a multimodal model processing process provided by an embodiment of the present application;

[0029] Figure 4 A schematic diagram of the structure of a data processing device based on sloppy prompts provided in one embodiment of the present application;

[0030] Figure 5 It is a block diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0031] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0032] The following describes the data processing method, device, electronic device and medium based on sloppy prompts in embodiments of the present application with reference to the accompanying drawings.

[0033] Figure 1 A flowchart of a data processing method based on scribbled prompts provided in an embodiment of the present application.

[0034] like Figure 1 As shown, the data processing method based on scribbled prompts includes the following steps:

[0035] Step 101, obtaining the scribbled prompt and the data to be processed corresponding to the data processing task.

[0036] The data processing task is a target recognition task, and the data to be processed is any one of image data, video data and point cloud data. Among them, the target recognition task includes tasks such as target detection tasks and target classification tasks.

[0037] A sloppy hint is one that is not completely accurate.

[0038] It should be noted that the sloppy prompt can refer to an inaccurate natural language prompt or an inaccurate visual prompt. The visual prompt refers to a specific visual element or mark added to the data to be processed, and the visual element can be a simple geometric shape, color mark, etc.

[0039] As an example but not limitation, assuming that the data processing task is to identify oranges in an image, the scribbled hint may be “identify oranges in an image” or a bounding box that does not completely surround the oranges.

[0040] Step 102: Generate a first feature vector based on the scribbled hint and the embedding vector and position vector corresponding to each mark in the data to be processed.

[0041] The scribbled prompt and the data to be processed constitute a multimodal input. After obtaining the multimodal input, the multimodal input can be vectorized to obtain an embedding vector of each token, and the multimodal input can be positionally encoded to obtain a position vector.

[0042] Add the position vector to the embedding vector to get the first eigenvector.

[0043] A tag refers to the smallest unit into which a multimodal input is segmented and encoded when processing the multimodal input. In an optional embodiment, for text, the tag may be at a word level; for an image, the tag may be at a pixel level or a pixel block level.

[0044] Optionally, the scribbled prompt and the data to be processed may be processed by a vector conversion model to obtain an embedding vector and a position vector, and then the position vector is added to the embedding vector to obtain a first feature vector.

[0045] Step 103: input the first feature vector into at least one attention residual model to perform feature conversion and enhancement to obtain a second feature vector.

[0046] When there are multiple attention residual models, the multiple attention residual models are connected in sequence, and the output of the previous attention residual model is the input of the next attention residual model.

[0047] The attention residual model includes at least one attention residual block and at least one nonlinear transformation residual block connected in sequence.

[0048] The output of the last nonlinear transformation residual block is the second eigenvector.

[0049] The attention residual model combines the self-attention mechanism and the residual mechanism. The self-attention mechanism can dynamically assign different weights to different input features, allowing the model to better focus on key information, which helps capture key features at different positions in the first feature vector during processing; the residual mechanism can directly connect across layers, which not only provides a shortcut path for the gradient, making it easier to pass the gradient to the previous layer, alleviating the gradient vanishing and gradient exploding problems, but also helps the model learn features better, thereby further improving model performance.

[0050] After the first eigenvector is processed by the attention residual model, a new eigenvector, namely the second eigenvector, can be obtained.

[0051] When the scribbled prompts and the data to be processed are input together, the attention residual model can learn the correlation and complementarity between them through the self-attention mechanism and the residual mechanism, and can focus on the key information in the data to be processed and the key information in the scribbled prompts at the same time, to achieve more refined multimodal feature fusion. For example, it can refer to the content of the data to be processed, correctly understand the intention of the scribbled prompts, and not be misled by the wrong information in the scribbled prompts.

[0052] Step 104: Determine the task processing result corresponding to the data to be processed based on the second eigenvector.

[0053] Optionally, based on the second eigenvector, determining the task processing result corresponding to the data to be processed includes: performing a linear transformation on the second eigenvector to obtain a linear transformation vector, the size of the linear transformation vector being the same as the size of the vocabulary corresponding to the data processing task; converting the linear transformation vector into a probability distribution, and determining the task processing result based on the probability distribution.

[0054] In an optional embodiment, if the data processing task is a target detection task, the vocabulary corresponding to the data processing task can be composed of various targets; if the data processing task is a target classification task, the vocabulary corresponding to the data processing task can be composed of target categories.

[0055] As an example but not limitation, the second eigenvector may be linearly transformed through a linear layer, and the linearly transformed vector may be converted into a probability distribution through a normalized exponential function (Softmax) layer.

[0056] As an example but not limitation, assuming that the data to be processed is an image, if the data processing task is a target detection task, the target corresponding to the maximum probability is the target included in the image; if the data processing task is a target classification task, the target category corresponding to the maximum probability is the category of the target included in the image.

[0057] In this embodiment, the scribbled prompt and the data to be processed corresponding to the data processing task are obtained; based on the embedding vectors and position vectors corresponding to each mark in the scribbled prompt and the data to be processed, a first feature vector is generated; the first feature vector is input into the attention residual model, and feature conversion and enhancement are performed to obtain a second feature vector; based on the second feature vector, the task processing result corresponding to the data to be processed is determined. By processing the scribbled prompt and the data to be processed through the attention residual model, the content of the data to be processed can be used to correctly understand the intention of the scribbled prompt, and the erroneous information in the scribbled prompt will not be misled, thereby ensuring the accuracy of the task processing result.

[0058] This embodiment provides another data processing method based on scribbled prompts. Figure 2 A flowchart of a data processing method based on scribbled prompts provided in an embodiment of the present application.

[0059] like Figure 2 As shown, the data processing method based on the scribbled prompt may include the following steps:

[0060] Step 201, obtaining the scribbled prompt and the data to be processed corresponding to the data processing task.

[0061] Step 202: Generate a first feature vector based on the scribbled hint and the embedding vector and position vector corresponding to each mark in the data to be processed.

[0062] The relevant contents in step 201 and step 202 can refer to the relevant description in step 101 and step 102, which will not be repeated here.

[0063] Step 203: Input the first feature vector into the first attention residual block in the first attention residual model to perform feature conversion and enhancement based on the self-attention mechanism and the residual mechanism to obtain an intermediate feature vector output by the first attention residual block.

[0064] The self-attention mechanism can refer to a Multi-Head Self-Attention mechanism or a Masked Multi-Head Self-Attention mechanism.

[0065] In an optional embodiment, the first feature vector is input into the first attention residual block to perform feature conversion and enhancement based on the self-attention mechanism and the residual mechanism to obtain an intermediate feature vector output by the first attention residual block, including: inputting the first feature vector into the first attention residual block to obtain the attention weight vector corresponding to the first feature vector based on the self-attention mechanism; obtaining the third feature vector based on the attention weight vector and the first feature vector; and performing a residual connection between the first feature vector and the third feature vector to obtain the intermediate feature vector output by the first attention residual block.

[0066] Performing residual connection on the first eigenvector and the third eigenvector may refer to performing feature fusion on the first eigenvector and the third eigenvector.

[0067] It should be noted that the intermediate feature vector output by each attention residual block may refer to a vector after layer normalization.

[0068] Combined with the data to be processed, the residual attention model can infer the intention of the scribbled prompt through contextual information. In addition, the multi-head self-attention mechanism enables the residual attention model to focus on the key parts of the scribbled prompt and ignore the wrong information, so as to understand the intention of the scribbled prompt more accurately.

[0069] If the attention residual model includes multiple attention residual blocks, the intermediate feature vector output by the first attention residual block is the input of the next attention residual block. It should be noted that the processing flow of the other attention residual blocks on their input is the same as that of the first attention residual block, which will not be repeated here.

[0070] Step 204, input the intermediate feature vector output by the last attention residual block into the first nonlinear transformation residual block, so as to perform feature conversion and enhancement based on the nonlinear transformation mechanism and the residual mechanism, and obtain the intermediate feature vector output by the first nonlinear transformation residual block.

[0071] The nonlinear transformation residual block includes a feed-forward neural network. As an example and not a limitation, the feed-forward neural network includes two linear transformation layers and a nonlinear activation function such as a ReLU activation function.

[0072] Inputting the intermediate feature vector output by the last attention residual block into the first nonlinear transformation residual block to perform feature conversion and enhancement based on the nonlinear transformation mechanism and the residual mechanism to obtain the intermediate feature vector output by the first nonlinear transformation residual block, including: inputting the intermediate feature vector output by the last attention residual block into the first nonlinear transformation residual block to perform nonlinear transformation on the intermediate feature vector based on a feedforward neural network to obtain a fourth feature vector; performing a residual connection between the intermediate feature vector output by the last attention residual block and the fourth feature vector to obtain the intermediate feature vector output by the first nonlinear transformation residual block.

[0073] It should be noted that the intermediate feature vector output by each nonlinear transformation residual block may refer to a vector after layer normalization.

[0074] By performing a nonlinear transformation on the intermediate feature vector output by the last attention residual block, richer feature information can be extracted and mapped, enabling the attention residual model to handle nonlinear problems and enhancing the expressive power of the attention residual model.

[0075] If the attention residual model includes multiple nonlinear transformation residual blocks, the intermediate feature vector output by the first nonlinear transformation residual block is the input of the next nonlinear transformation residual block. It should be noted that the processing flow of other nonlinear transformation residual blocks on their input is the same as that of the first nonlinear transformation residual block, which will not be repeated here.

[0076] Step 205, when the number of attention residual models is one, the intermediate feature vector output by the last nonlinear transformation residual block is used as the second feature vector.

[0077] When the number of the attention residual model is one, the output of the attention residual model can be directly used as the second eigenvector, wherein the second eigenvector is a vector after layer normalization.

[0078] Step 206, when there are multiple attention residual models, the intermediate feature vector output by the last nonlinear transformation residual block is input into the next attention residual model connected in sequence, and the intermediate feature vector output by the last nonlinear transformation residual block in the last attention residual model is used as the second feature vector.

[0079] The processing process of each attention residual model is the same. The processing process of the remaining attention residual models can refer to the description of the processing process of the first attention residual model, which will not be repeated here.

[0080] Step 207: Determine the task processing result corresponding to the data to be processed based on the second eigenvector.

[0081] If the data processing task is a target recognition task, the task processing result may be the target's bounding box, classification label, features in the image, and other results.

[0082] In this embodiment, the first feature vector is input into the first attention residual block in the first attention residual model, so as to perform feature conversion and enhancement based on the self-attention mechanism and the residual mechanism, and obtain the intermediate feature vector output by the first attention residual block; the intermediate feature vector output by the last attention residual block is input into the first nonlinear transformation residual block, so as to perform feature conversion and enhancement based on the nonlinear transformation mechanism and the residual mechanism, and obtain the intermediate feature vector output by the first nonlinear transformation residual block; when the number of attention residual models is one, the intermediate feature vector output by the last nonlinear transformation residual block is used as the second feature vector; when the number of attention residual models is multiple, the intermediate feature vector output by the last nonlinear transformation residual block is input into the next attention residual model connected in sequence, and the intermediate feature vector output by the last nonlinear transformation residual block in the last attention residual model is used as the second feature vector; based on the second feature vector, the task processing result corresponding to the data to be processed is determined. Combining the attention residual block and the nonlinear transformation residual block can enhance the feature extraction and expression capabilities of the attention residual model, so that the attention residual model can pay more attention to the key information in the scribbled prompts and the data to be processed, correctly understand the intention of the scribbled prompts, and thus ensure the accuracy of the task processing results.

[0083] In an optional embodiment, the vector conversion model, at least one attention residual model, the linear layer and the Softmax layer in the aforementioned embodiment constitute a multimodal model.

[0084] See also Figure 3 , Figure 3 The following is a flowchart of the multimodal model processing process provided in the embodiment of the present application, as an example: Figure 3 The case where the multimodal model includes an attention residual model is shown in .

[0085] After obtaining the scribbled prompt and the data to be processed, the scribbled prompt and the data to be processed are input into the multimodal model for processing to obtain the task processing result output by the multimodal model.

[0086] Figure 3 The input sequence in is the scribbled prompt and the data to be processed, the decoder layer is the above-mentioned attention residual model, the input of the decoder layer is the first eigenvector, the residual connection and layer normalization represent Add&norm, the output of the decoder layer, that is, the output sequence, is the second eigenvector, and the final output is the task processing result.

[0087] In order to implement the above embodiment, the embodiment of the present application also proposes a data processing device based on sloppy prompts.

[0088] Figure 4 A schematic diagram of the structure of a data processing device based on scribbled prompts provided in an embodiment of the present application.

[0089] like Figure 4 As shown, the data processing device 400 based on the sloppy prompt includes:

[0090] An acquisition module 410 is used to acquire the scribbled prompt and the data to be processed corresponding to the data processing task;

[0091] A first processing module 420, configured to generate a first feature vector based on the scribbled hint and the embedding vector and the position vector corresponding to each mark in the data to be processed;

[0092] A second processing module 430 is used to input the first feature vector into at least one attention residual model, perform feature conversion and enhancement, and obtain a second feature vector;

[0093] The result determination module 440 is used to determine the task processing result corresponding to the data to be processed based on the second feature vector.

[0094] Optionally, the attention residual model includes at least one attention residual block and at least one nonlinear transformation residual block connected in sequence, and the second processing module 430 is used to:

[0095] Inputting the first feature vector into a first attention residual block in the first attention residual model to perform feature conversion and enhancement based on a self-attention mechanism and a residual mechanism to obtain an intermediate feature vector output by the first attention residual block;

[0096] The intermediate feature vector output by the last attention residual block is input into the first nonlinear transformation residual block to perform feature conversion and enhancement based on the nonlinear transformation mechanism and the residual mechanism to obtain the intermediate feature vector output by the first nonlinear transformation residual block;

[0097] When the number of attention residual models is one, the intermediate feature vector output by the last nonlinear transformation residual block is used as the second feature vector;

[0098] When there are multiple attention residual models, the intermediate feature vector output by the last nonlinear transformation residual block is input into the next attention residual model connected in sequence, and the intermediate feature vector output by the last nonlinear transformation residual block in the last attention residual model is used as the second feature vector.

[0099] Optionally, the second processing module 430 is used to:

[0100] Inputting the first feature vector into the first attention residual block in the first attention residual model to obtain an attention weight vector corresponding to the first feature vector based on the self-attention mechanism;

[0101] Based on the attention weight vector and the first eigenvector, obtaining a third eigenvector;

[0102] The first eigenvector and the third eigenvector are residually connected to obtain the intermediate eigenvector output by the first attention residual block.

[0103] Optionally, the nonlinear transformation residual block includes a feedforward neural network, and the second processing module 430 is used to:

[0104] Input the intermediate feature vector output by the last attention residual block into the first nonlinear transformation residual block to perform nonlinear transformation on the intermediate feature vector based on a feedforward neural network to obtain a fourth feature vector;

[0105] The intermediate feature vector output by the last attention residual block is residually connected with the fourth feature vector to obtain the intermediate feature vector output by the first nonlinear transformation residual block.

[0106] Optionally, the result determination module 440 is used to:

[0107] Performing a linear transformation on the second eigenvector to obtain a linear transformation vector, wherein the size of the linear transformation vector is the same as the size of the vocabulary corresponding to the data processing task;

[0108] The linear transformation vector is converted into a probability distribution, and based on the probability distribution, the task processing result is determined.

[0109] Optionally, the data processing task is a target recognition task, and the data to be processed is any one of image data, video data and point cloud data.

[0110] It should be noted that the aforementioned explanation of the embodiment of the data processing method based on scribbled prompts is also applicable to the data processing device based on scribbled prompts of this embodiment, and will not be repeated here.

[0111] Figure 5A schematic diagram of the structure of an electronic device provided for an embodiment of the present application. Among them, the electronic device 500 in this embodiment is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0112] like Figure 5 As shown, the electronic device 500 includes:

[0113] The memory 501 and the processor 502, a bus 503 connecting different components (including the memory 501 and the processor 502), the memory 501 stores a computer program, and when the processor 502 executes the program, the data processing method based on the sloppy prompt of the embodiment of the present application is implemented.

[0114] Bus 503 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus structures. For example, these architectures include but are not limited to Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus and Peripheral Component Interconnect (PCI) bus.

[0115] The electronic device 500 typically includes a variety of electronic device readable media. These media can be any available media that can be accessed by the electronic device 500, including volatile and non-volatile media, removable and non-removable media.

[0116] The memory 501 may also include computer system readable media in the form of volatile memory, such as random access memory (RAM) 504 and / or cache memory 505. The electronic device 500 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 506 may be used to read and write non-removable, non-volatile magnetic media ( Figure 5 not shown, usually called a "hard drive"). Although Figure 5Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to the bus 503 via one or more data medium interfaces. The memory 501 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of each embodiment of the present application.

[0117] A program / utility 508 having a set (at least one) of program modules 507 may be stored, for example, in the memory 501, such program modules 507 including but not limited to an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The program modules 507 generally perform the functions and / or methods of the embodiments described herein.

[0118] The electronic device 500 may also communicate with one or more external devices 509 (e.g., keyboard, pointing device, display 511, etc.), one or more devices that enable a user to interact with the electronic device 500, and / or any device that enables the electronic device 500 to communicate with one or more other computing devices (e.g., network card, modem, etc.). Such communication may be performed through an input / output (I / O) interface 512. Furthermore, the electronic device 500 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 513. Figure 5 As shown, the network adapter 513 communicates with other modules of the electronic device 500 via the bus 503. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the electronic device 500, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0119] The processor 502 executes various functional applications and data processing by running the programs stored in the memory 501 .

[0120] It should be noted that the implementation process and technical principles of the electronic device of this embodiment refer to the aforementioned explanation of the data processing method based on scribbled prompts in the embodiment of the present application, and will not be repeated here.

[0121] In order to implement the above embodiments, the present application also proposes a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the methods provided by the above embodiments.

[0122] In order to implement the above embodiments, the present application also proposes a computer program product, including a computer program, which implements the methods provided by the above embodiments when executed by a processor.

[0123] The collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in this application are in compliance with relevant laws and regulations and do not violate public order and good morals.

[0124] It should be noted that personal information from users should be collected for legitimate and reasonable purposes and should not be shared or sold outside of these legitimate uses. In addition, such collection / sharing should be carried out after receiving the user's informed consent, including but not limited to notifying the user to read the user agreement / user notice and sign the agreement / authorization including authorization of relevant user information before the user uses the function. In addition, any necessary steps should be taken to protect and safeguard access to such personal information data and ensure that others who have access to personal information data comply with its privacy policy and procedures.

[0125] The present application is expected to provide an implementation scheme for users to selectively block the use or access of personal information data. That is, the present disclosure is expected to provide hardware and / or software to prevent or block access to such personal information data. Once the personal information data is no longer needed, the risk can be minimized by limiting data collection and deleting the data. In addition, when applicable, such personal information is de-identified to protect the privacy of the user.

[0126] In the description of the aforementioned embodiments, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0127] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0128] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0129] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples (non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0130] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0131] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0132] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0133] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A data processing method based on scribbling prompts, characterized in that: The following steps are involved: Obtaining scribbled prompts and data to be processed corresponding to a data processing task, wherein the data processing task is a target recognition task, and the data to be processed is any one of image data, video data, and point cloud data, wherein the scribbled prompts include inaccurate natural language prompts and inaccurate visual prompts, and the visual prompts refer to specific visual elements or marks added to the data to be processed; Based on the embedding vectors and position vectors corresponding to each mark in the scribbled prompt and the data to be processed, a first feature vector is generated, including: the scribbled prompt and the data to be processed constitute a multimodal input, after obtaining the multimodal input, vector conversion is performed on the multimodal input to obtain an embedding vector of each mark, and position encoding is performed on the multimodal input to obtain a position vector, and the position vector is added to the embedding vector to obtain the first feature vector, wherein the mark refers to the smallest unit into which the multimodal input is divided and encoded when processing the multimodal input; Inputting the first feature vector into at least one attention residual model to perform feature conversion and enhancement to obtain a second feature vector; Determining the task processing result corresponding to the to-be-processed data based on the second feature vector, including: performing a linear transformation on the second feature vector to obtain a linear transformation vector, wherein the size of the linear transformation vector is the same as the size of the vocabulary corresponding to the data processing task; converting the linear transformation vector into a probability distribution, and determining the task processing result based on the probability distribution; The attention residual model includes at least one attention residual block and at least one nonlinear transformation residual block connected in sequence, and the first feature vector is input into at least one attention residual model to perform feature conversion and enhancement to obtain a second feature vector, including: Inputting the first feature vector into the first attention residual block in the first attention residual model to perform feature conversion and enhancement based on the self-attention mechanism and the residual mechanism to obtain an intermediate feature vector output by the first attention residual block. If the attention residual model includes multiple attention residual blocks, the intermediate feature vector output by the first attention residual block is the input of the next attention residual block. Inputting the intermediate feature vector output by the last attention residual block into the first nonlinear transformation residual block to perform feature conversion and enhancement based on the nonlinear transformation mechanism and the residual mechanism, thereby obtaining the intermediate feature vector output by the first nonlinear transformation residual block; When the number of the attention residual model is one, taking the intermediate feature vector output by the last nonlinear transformation residual block as the second feature vector; When there are multiple attention residual models, the intermediate feature vector output by the last nonlinear transformation residual block is input into the next attention residual model connected in sequence, and the intermediate feature vector output by the last nonlinear transformation residual block in the last attention residual model is used as the second feature vector.

2. The method according to claim 1, characterized in that The step of inputting the first feature vector into the first attention residual block in the first attention residual model to perform feature conversion and enhancement based on the self-attention mechanism and the residual mechanism to obtain an intermediate feature vector output by the first attention residual block includes: Inputting the first feature vector into the first attention residual block in the first attention residual model to obtain an attention weight vector corresponding to the first feature vector based on the self-attention mechanism; Based on the attention weight vector and the first feature vector, obtaining a third feature vector; The first feature vector and the third feature vector are residually connected to obtain an intermediate feature vector output by the first attention residual block.

3. The method according to claim 1, characterized in that The nonlinear transformation residual block includes a feedforward neural network, and the intermediate feature vector output by the last attention residual block is input into the first nonlinear transformation residual block to perform feature conversion and enhancement based on the nonlinear transformation mechanism and the residual mechanism to obtain the intermediate feature vector output by the first nonlinear transformation residual block, including: Inputting the intermediate feature vector output by the last attention residual block into the first nonlinear transformation residual block to perform nonlinear transformation on the intermediate feature vector based on the feedforward neural network to obtain a fourth feature vector; The intermediate feature vector output by the last attention residual block and the fourth feature vector are residually connected to obtain the intermediate feature vector output by the first nonlinear transformation residual block.

4. A data processing device based on scribbling prompts, characterized in that: include: an acquisition module, used for acquiring scribbled prompts and to-be-processed data corresponding to a data processing task, wherein the data processing task is a target recognition task, and the to-be-processed data is any one of image data, video data and point cloud data, wherein the scribbled prompts include inaccurate natural language prompts and inaccurate visual prompts, and the visual prompts refer to specific visual elements or marks added to the to-be-processed data; A first processing module, for generating a first feature vector based on the embedding vectors and position vectors corresponding to each mark in the scribbled prompt and the data to be processed, comprising: the scribbled prompt and the data to be processed constitute a multimodal input, after obtaining the multimodal input, performing vector conversion on the multimodal input to obtain an embedding vector of each mark, and performing position encoding on the multimodal input to obtain a position vector, adding the position vector to the embedding vector to obtain the first feature vector, wherein the mark refers to the smallest unit into which the multimodal input is divided and encoded when processing the multimodal input; A second processing module, configured to input the first feature vector into at least one attention residual model, perform feature conversion and enhancement, and obtain a second feature vector; A result determination module is used to determine the task processing result corresponding to the to-be-processed data based on the second feature vector, including: performing a linear transformation on the second feature vector to obtain a linear transformation vector, wherein the size of the linear transformation vector is the same as the size of the vocabulary corresponding to the data processing task; converting the linear transformation vector into a probability distribution, and determining the task processing result based on the probability distribution; The attention residual model includes at least one attention residual block and at least one nonlinear transformation residual block connected in sequence, and the first feature vector is input into at least one attention residual model to perform feature conversion and enhancement to obtain a second feature vector, including: Inputting the first feature vector into the first attention residual block in the first attention residual model to perform feature conversion and enhancement based on the self-attention mechanism and the residual mechanism to obtain an intermediate feature vector output by the first attention residual block. If the attention residual model includes multiple attention residual blocks, the intermediate feature vector output by the first attention residual block is the input of the next attention residual block. Inputting the intermediate feature vector output by the last attention residual block into the first nonlinear transformation residual block to perform feature conversion and enhancement based on the nonlinear transformation mechanism and the residual mechanism, thereby obtaining the intermediate feature vector output by the first nonlinear transformation residual block; When the number of the attention residual model is one, taking the intermediate feature vector output by the last nonlinear transformation residual block as the second feature vector; When there are multiple attention residual models, the intermediate feature vector output by the last nonlinear transformation residual block is input into the next attention residual model connected in sequence, and the intermediate feature vector output by the last nonlinear transformation residual block in the last attention residual model is used as the second feature vector.

5. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 3 when executed by a processor.

7. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 3 when being executed by a processor.

Images