Data processing method and apparatus based on sloppy prompt, and electronic device and medium

Abstract

Provided in the present application are a data processing method and apparatus based on a sloppy prompt, and an electronic device and a medium. The method comprises: acquiring a sloppy prompt and data to be processed which correspond to a data processing task; on the basis of the sloppy prompt, and an embedding vector and a position vector which correspond to each mark in said data, generating a first feature vector; inputting the first feature vector into at least one attention residual model, and performing feature conversion and enhancement to obtain a second feature vector; and on the basis of the second feature vector, determining a task processing result corresponding to said data. A sloppy prompt and data to be processed are processed by means of an attention residual model, such that the content of said data can be used for reference to correctly understand the intent of the sloppy prompt, without being misled by erroneous information in the sloppy prompt, thereby ensuring the accuracy of a task processing result.

Description

Data processing method and device based on messy prompt, electronic device and medium

[0001] Cross-reference to related applications

[0002] The present application claims priority to the Chinese patent application No. 202411119630.0, filed on August 15, 2024, entitled "Data processing method and device based on messy prompt, electronic device and medium". TECHNICAL FIELD

[0003] The present application relates to the technical field of data processing, and in particular to a data processing method and device based on messy prompt, an electronic device and a medium. BACKGROUND

[0004] The data processing task based on prompt, such as the object detection task and the object classification task, is a technology of guiding the computer vision model to perform a specific task by using a natural language prompt or a visual prompt. The accuracy of the task processing result is related to the accuracy of the prompt information. If the prompt is a messy prompt, that is, the prompt is not completely accurate, the task processing result may be inaccurate. Therefore, how to ensure the accuracy of the task processing result under the condition that the prompt is not completely accurate is a technical problem to be solved. SUMMARY

[0005] The present application aims to at least solve one of the technical problems in the related art to some extent.

[0006] To this end, the first object of the present application is to provide a data processing method based on a messy prompt.

[0007] The second object of the present application is to provide a data processing device based on a messy prompt.

[0008] The third object of the present application is to provide an electronic device.

[0009] The fourth object of the present application is to provide a computer readable storage medium.

[0010] The fifth object of the present application is to provide a computer program product.

[0011] To achieve the above objects, the first aspect of the present application provides a data processing method based on a messy prompt, comprising:

[0012] obtaining a messy prompt corresponding to a data processing task and data to be processed;

[0013] generating a first feature vector based on the embedding vector and the position vector corresponding to each mark in the messy prompt and the data to be processed;

[0014] input the first feature vector into at least one attention residual model to perform feature conversion and enhancement, to obtain a second feature vector;

[0015] determine a task processing result corresponding to the to-be-processed data based on the second feature vector.

[0016] To achieve the above object, the second aspect of the present application provides a data processing device based on a messy prompt, comprising:

[0017] an acquisition module, configured to acquire a messy prompt corresponding to a data processing task and to-be-processed data;

[0018] a first processing module, configured to generate a first feature vector based on an embedding vector and a position vector corresponding to each mark in the messy prompt and the to-be-processed data;

[0019] a second processing module, configured to input the first feature vector into at least one attention residual model to perform feature conversion and enhancement, to obtain a second feature vector;

[0020] a result determination module, configured to determine a task processing result corresponding to the to-be-processed data based on the second feature vector.

[0021] To achieve the above object, the third aspect of the present application provides an electronic device, comprising a processor and a memory connected with the processor; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory to implement the data processing method based on a messy prompt according to the first aspect of the present application.

[0022] To achieve the above object, the fourth aspect of the present application provides a computer readable storage medium, which stores computer execution instructions; when the computer execution instructions are executed by a processor, the computer execution instructions are used to implement the data processing method based on a messy prompt according to the first aspect of the present application.

[0023] To achieve the above object, the fifth aspect of the present application provides a computer program product, comprising a computer program; when the computer program is executed by a processor, the computer program implements the data processing method based on a messy prompt according to the first aspect of the present application.

[0024] The technical scheme provided by the present application at least brings the following beneficial effects:

[0025] The application obtains a messy prompt corresponding to a data processing task and to-be-processed data; generates a first feature vector based on an embedding vector and a position vector corresponding to each mark in the messy prompt and the to-be-processed data; inputs the first feature vector into at least one attention residual model to perform feature conversion and enhancement, and obtains a second feature vector; and determines a task processing result corresponding to the to-be-processed data based on the second feature vector. The messy prompt and the to-be-processed data are processed by the attention residual model, the content of the to-be-processed data can be referred to, the intention of the messy prompt can be correctly understood, and the to-be-processed data is not misled by the error information in the messy prompt, thereby ensuring the accuracy of the task processing result.

[0026] Additional aspects and advantages of the application will be set forth in part in the description which follows, and in part will become apparent to those skilled in the art upon examination of the following and / or can be learned by practice of the application. BRIEF DESCRIPTION OF DRAWINGS

[0027] The above and / or additional aspects and advantages of the application will become apparent and be readily appreciated from the following description, taken in conjunction with the accompanying drawings, in which:

[0028] FIG. 1 is a flowchart of a data processing method based on a messy prompt according to an embodiment of the present application;

[0029] FIG. 2 is a flowchart of a data processing method based on a messy prompt according to another embodiment of the present application;

[0030] FIG. 3 is a flowchart of a multi-modal model processing process according to an embodiment of the present application;

[0031] FIG. 4 is a structural diagram of a data processing device based on a messy prompt according to an embodiment of the present application;

[0032] FIG. 5 is a block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0033] Embodiments of the present application are described in detail below, examples of which are shown in the accompanying drawings, in which the same or similar reference numerals represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by reference to the accompanying drawings are exemplary and are intended to explain the present application, and cannot be understood as limiting the present application.

[0034] The data processing method, device, electronic device and medium based on a messy prompt according to the embodiments of the present application are described below with reference to the accompanying drawings.

[0035] FIG. 1 is a flowchart of a data processing method based on a messy prompt according to an embodiment of the present application.

[0036] As shown in FIG. 1, the data processing method based on the rough prompt includes the following steps:

[0037] In step 101, a rough prompt corresponding to a data processing task and to-be-processed data are obtained.

[0038] The data processing task is a target recognition task, and the to-be-processed data is any one of image data, video data, and point cloud data. The target recognition task includes a target detection task and a target classification task.

[0039] The rough prompt refers to an inaccurate prompt.

[0040] It should be noted that the rough prompt can refer to an inaccurate natural language prompt or an inaccurate visual prompt. The visual prompt refers to a specific visual element or mark added in the to-be-processed data. The visual element can be a simple geometric shape, a color mark, or the like.

[0041] By way of example but not limitation, assuming that the data processing task is to identify an orange in an image, the rough prompt can be "identify an orange in an image" or a bounding box that does not completely surround the orange.

[0042] In step 102, a first feature vector is generated based on an embedding vector and a position vector corresponding to each mark in the rough prompt and the to-be-processed data.

[0043] The rough prompt and the to-be-processed data constitute a multi-modal input. After obtaining the multi-modal input, the multi-modal input can be vector-converted to obtain an embedding vector of each mark (Token), and the multi-modal input can be position-encoded to obtain a position vector.

[0044] The position vector is added to the embedding vector to obtain the first feature vector.

[0045] The mark refers to the smallest unit into which the multi-modal input is divided and encoded when processing the multi-modal input. In an optional embodiment, for text, the mark can be at the word level; for an image, the mark can be at the pixel level or at the pixel block level.

[0046] Optionally, the rough prompt and the to-be-processed data can be processed by a vector conversion model to obtain the embedding vector and the position vector, and then the position vector is added to the embedding vector to obtain the first feature vector.

[0047] In step 103, the first feature vector is input into at least one attention residual model for feature conversion and enhancement to obtain a second feature vector.

[0048] When the number of attention residual models is multiple, the multiple attention residual models are sequentially connected, and the output of a previous attention residual model is the input of a next attention residual model.

[0049] The attention residual model comprises at least one attention residual block and at least one nonlinear transformation residual block which are sequentially connected.

[0050] The output of the last nonlinear transformation residual block is the second feature vector.

[0051] The attention residual model combines the self-attention mechanism and the residual mechanism. The self-attention mechanism can dynamically assign different weights to different input features, so that the model can better focus on key information, which helps to capture key features at different positions in the first feature vector during processing. The residual mechanism can provide a shortcut path for the gradient, so that the gradient can more easily pass to the previous layer, alleviating the problems of gradient vanishing and gradient explosion, and can also help the model to better learn features, thereby further improving the performance of the model.

[0052] After the first feature vector is processed by the attention residual model, a new feature vector, i.e., the second feature vector, can be obtained.

[0053] When the rough prompt and the to-be-processed data are input together, the attention residual model can learn the relevance and complementarity between them through the self-attention mechanism and the residual mechanism, and can simultaneously focus on the key information in the to-be-processed data and the key information in the rough prompt, achieving more fine-grained multi-modal feature fusion, such as correctly understanding the intention of the rough prompt by referring to the content of the to-be-processed data, and not being misled by the false information in the rough prompt.

[0054] In step 104, based on the second feature vector, a task processing result corresponding to the to-be-processed data is determined.

[0055] Optionally, based on the second feature vector, the task processing result corresponding to the to-be-processed data is determined, including: performing linear transformation on the second feature vector to obtain a linear transformation vector, the size of the linear transformation vector being the same as the size of a vocabulary corresponding to the data processing task; converting the linear transformation vector into a probability distribution, and determining the task processing result based on the probability distribution.

[0056] In an optional embodiment, if the data processing task is an object detection task, the vocabulary corresponding to the data processing task can be composed of various objects; if the data processing task is an object classification task, the vocabulary corresponding to the data processing task can be composed of object categories.

[0057] As an example but not limitation, the second feature vector can be linearly transformed by a linear layer, and the linearly transformed vector can be converted into a probability distribution by a normalization exponential function (Softmax) layer.

[0058] As an example but not limitation, assuming that the data to be processed is an image, if the data processing task is an object detection task, the object corresponding to the maximum probability is the object included in the image; if the data processing task is an object classification task, the object category corresponding to the maximum probability is the category of the object included in the image.

[0059] In this embodiment, the scribble prompt corresponding to the data processing task and the data to be processed are obtained; based on the embedding vectors and the position vectors corresponding to each mark in the scribble prompt and the data to be processed, a first feature vector is generated; the first feature vector is input into an attention residual model for feature conversion and enhancement to obtain a second feature vector; and based on the second feature vector, a task processing result corresponding to the data to be processed is determined. By processing the scribble prompt and the data to be processed through the attention residual model, the content of the data to be processed can be referred to, the intention of the scribble prompt can be correctly understood, and the data processing result can be ensured to be accurate.

[0060] This embodiment provides another data processing method based on a scribble prompt. FIG. 2 is a flowchart of a data processing method based on a scribble prompt provided by an embodiment of the present application.

[0061] As shown in FIG. 2, the data processing method based on a scribble prompt can include the following steps:

[0062] Step 201: obtaining a scribble prompt corresponding to a data processing task and data to be processed.

[0063] Step 202: based on embedding vectors and position vectors corresponding to each mark in the scribble prompt and the data to be processed, a first feature vector is generated.

[0064] The related content in steps 201 and 202 can refer to the related description in steps 101 and 102, which will not be repeated here.

[0065] Step 203: inputting the first feature vector into a first attention residual block of a first attention residual model to perform feature conversion and enhancement based on a self-attention mechanism and a residual mechanism to obtain an intermediate feature vector output by the first attention residual block.

[0066] The self-attention mechanism can be a multi-head self-attention (Multi-Head Self-Attention) mechanism, or a masked multi-head self-attention (Masked Multi-Head Self-Attention) mechanism.

[0067] In an optional embodiment, the first feature vector is input into the first attention residual block to perform feature conversion and enhancement based on a self-attention mechanism and a residual mechanism, to obtain an intermediate feature vector output by the first attention residual block, including: inputting the first feature vector into the first attention residual block to obtain an attention weight vector corresponding to the first feature vector based on the self-attention mechanism; obtaining a third feature vector based on the attention weight vector and the first feature vector; performing residual connection on the first feature vector and the third feature vector to obtain the intermediate feature vector output by the first attention residual block.

[0068] The residual connection on the first feature vector and the third feature vector can mean feature fusion on the first feature vector and the third feature vector.

[0069] It should be noted that the intermediate feature vector output by each attention residual block can mean a vector after layer normalization.

[0070] In combination with the data to be processed, the attention residual model can infer the intent of the messy prompt through the context information. In addition, the multi-head self-attention mechanism enables the attention residual model to focus on the key part of the messy prompt and ignore the error information, so as to more accurately understand the intent of the messy prompt.

[0071] If the attention residual model includes multiple attention residual blocks, the intermediate feature vector output by the first attention residual block is the input of the next attention residual block. It should be noted that the processing procedure of other attention residual blocks on their inputs is the same as that of the first attention residual block, which will not be described here.

[0072] In step 204, the intermediate feature vector output by the last attention residual block is input into the first nonlinear transformation residual block to perform feature conversion and enhancement based on a nonlinear transformation mechanism and a residual mechanism, to obtain an intermediate feature vector output by the first nonlinear transformation residual block.

[0073] The nonlinear transformation residual block includes a feed-forward neural network. As an example but not limitation, the feed-forward neural network includes two linear transformation layers and a nonlinear activation function such as a ReLU activation function.

[0074] input the intermediate feature vector output by the last attention residual block into the first nonlinear transformation residual block to perform feature conversion and enhancement based on a nonlinear transformation mechanism and a residual mechanism, to obtain an intermediate feature vector output by the first nonlinear transformation residual block, including: inputting the intermediate feature vector output by the last attention residual block into the first nonlinear transformation residual block to perform nonlinear transformation on the intermediate feature vector based on a feedforward neural network, to obtain a fourth feature vector; and performing residual connection on the intermediate feature vector output by the last attention residual block and the fourth feature vector, to obtain the intermediate feature vector output by the first nonlinear transformation residual block.

[0075] It should be noted that the intermediate feature vector output by each nonlinear transformation residual block can be a vector after layer normalization.

[0076] Performing nonlinear transformation on the intermediate feature vector output by the last attention residual block can extract and map richer feature information, so that the attention residual model can process nonlinear problems and enhance the expression ability of the attention residual model.

[0077] If the attention residual model includes multiple nonlinear transformation residual blocks, the intermediate feature vector output by the first nonlinear transformation residual block is the input of the next nonlinear transformation residual block. It should be noted that the processing procedure of other nonlinear transformation residual blocks on their inputs is the same as that of the first nonlinear transformation residual block, which will not be described here.

[0078] In step 205, in the case where the number of attention residual models is one, the intermediate feature vector output by the last nonlinear transformation residual block is taken as the second feature vector.

[0079] In the case where the number of attention residual models is one, the output of the attention residual model can be directly taken as the second feature vector. The second feature vector is a vector after layer normalization.

[0080] In step 206, in the case where the number of attention residual models is multiple, the intermediate feature vector output by the last nonlinear transformation residual block is input into the next attention residual model connected in sequence, and the intermediate feature vector output by the last nonlinear transformation residual block in the last attention residual model is taken as the second feature vector.

[0081] The processing procedure of each attention residual model is the same, and the processing procedure of the remaining attention residual models can be specifically referred to the description of the processing procedure of the first attention residual model, which will not be described here.

[0082] In step 207, based on the second feature vector, a task processing result corresponding to the to-be-processed data is determined.

[0083] If the data processing task is a target recognition task, the task processing result can be a bounding box of the target, a classification label, and a feature in the image, etc.

[0084] In this embodiment, the first feature vector is input into the first attention residual block of the first attention residual model to perform feature conversion and enhancement based on the self-attention mechanism and the residual mechanism, to obtain the intermediate feature vector output by the first attention residual block; the intermediate feature vector output by the last attention residual block is input into the first nonlinear transformation residual block to perform feature conversion and enhancement based on the nonlinear transformation mechanism and the residual mechanism, to obtain the intermediate feature vector output by the first nonlinear transformation residual block; in the case where the number of attention residual models is one, the intermediate feature vector output by the last nonlinear transformation residual block is taken as the second feature vector; in the case where the number of attention residual models is multiple, the intermediate feature vector output by the last nonlinear transformation residual block is input into the next attention residual model connected in turn, and the intermediate feature vector output by the last nonlinear transformation residual block in the last attention residual model is taken as the second feature vector; based on the second feature vector, the task processing result corresponding to the to-be-processed data is determined. By combining the attention residual block and the nonlinear transformation residual block, the feature extraction and expression capability of the attention residual model can be enhanced, so that the attention residual model pays more attention to the key information in the rough prompt and the to-be-processed data, correctly understands the intention of the rough prompt, and thus ensures the accuracy of the task processing result.

[0085] In an optional embodiment, the vector conversion model, the at least one attention residual model, the linear layer, and the Softmax layer in the foregoing embodiments constitute a multi-modal model.

[0086] Referring to FIG. 3, FIG. 3 is a flowchart of a multi-modal model processing process provided by an embodiment of the present application. As an example, FIG. 3 shows the case where the multi-modal model includes one attention residual model.

[0087] After the rough prompt and the to-be-processed data are obtained, the rough prompt and the to-be-processed data are input into the multi-modal model for processing, to obtain the task processing result output by the multi-modal model.

[0088] The input sequence in FIG. 3 is the rough prompt and the to-be-processed data, the decoder layer is the attention residual model described above, the input of the decoder layer is the first feature vector, the residual connection and layer normalization are represented as Add&norm, the output of the decoder layer is also the output sequence, which is the second feature vector, and the final output is the task processing result.

[0089] To implement the foregoing embodiments, an embodiment of the present application further provides a data processing apparatus based on a rough prompt.

[0090] Fig. 4 is a structural schematic diagram of a data processing apparatus based on a messy prompt according to an embodiment of the present application.

[0091] As shown in Fig. 4, the data processing apparatus 400 based on the messy prompt includes:

[0092] The obtaining module 410 is configured to obtain a messy prompt corresponding to a data processing task and to-be-processed data.

[0093] The first processing module 420 is configured to generate a first feature vector based on the embedding vectors and the position vectors corresponding to the respective tokens in the messy prompt and the to-be-processed data.

[0094] The second processing module 430 is configured to input the first feature vector into at least one attention residual model to perform feature conversion and enhancement, and obtain a second feature vector.

[0095] The result determining module 440 is configured to determine a task processing result corresponding to the to-be-processed data based on the second feature vector.

[0096] Optionally, the attention residual model includes at least one attention residual block and at least one nonlinear transformation residual block connected in sequence, and the second processing module 430 is configured to:

[0097] input the first feature vector into a first attention residual block in a first attention residual model to perform feature conversion and enhancement based on a self-attention mechanism and a residual mechanism, and obtain an intermediate feature vector output by the first attention residual block;

[0098] input the intermediate feature vector output by the last attention residual block into a first nonlinear transformation residual block to perform feature conversion and enhancement based on a nonlinear transformation mechanism and a residual mechanism, and obtain an intermediate feature vector output by the first nonlinear transformation residual block;

[0099] in a case where the number of attention residual models is one, the intermediate feature vector output by the last nonlinear transformation residual block is taken as the second feature vector;

[0100] in a case where the number of attention residual models is multiple, the intermediate feature vector output by the last nonlinear transformation residual block is input into a next attention residual model connected in sequence, and the intermediate feature vector output by the last nonlinear transformation residual block in the last attention residual model is taken as the second feature vector.

[0101] Optionally, the second processing module 430 is configured to:

[0102] input the first feature vector into a first attention residual block in a first attention residual model to obtain an attention weight vector corresponding to the first feature vector based on a self-attention mechanism;

[0103] obtaining a third feature vector based on the attention weight vector and the first feature vector;

[0104] performing residual connection on the first feature vector and the third feature vector to obtain an intermediate feature vector output by the first attention residual block.

[0105] Optionally, the nonlinear transformation residual block comprises a feedforward neural network, and the second processing module 430 is configured to:

[0106] inputting the intermediate feature vector output by the last attention residual block into the first nonlinear transformation residual block to perform nonlinear transformation on the intermediate feature vector based on the feedforward neural network to obtain a fourth feature vector;

[0107] performing residual connection on the intermediate feature vector output by the last attention residual block and the fourth feature vector to obtain an intermediate feature vector output by the first nonlinear transformation residual block.

[0108] Optionally, the result determining module 440 is configured to:

[0109] performing linear transformation on the second feature vector to obtain a linear transformation vector, the linear transformation vector having the same size as a size of a vocabulary corresponding to the data processing task;

[0110] converting the linear transformation vector into a probability distribution, and determining the task processing result based on the probability distribution.

[0111] Optionally, the data processing task is a target recognition task, and the to-be-processed data is any one of image data, video data and point cloud data.

[0112] It should be noted that the foregoing explanation and description of the data processing method embodiment based on the scrawl prompt also applies to the data processing apparatus based on the scrawl prompt of this embodiment, which will not be described here.

[0113] FIG. 5 is a structural schematic diagram of an electronic device provided by an embodiment of the present application. The electronic device 500 in the embodiment is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular telephones, smart phones, wearable devices, and other similar computing devices. The components shown in the figures, their connections and relationships, and their functions, are merely examples and are not intended to limit the implementations of the present disclosure described and / or claimed in this document.

[0114] As shown in FIG. 5, the above-mentioned electronic device 500 comprises:

[0115] The memory 501 and the processor 502, the bus 503 connecting different components (including the memory 501 and the processor 502), the memory 501 stores a computer program, when the processor 502 executes the program, the data processing method based on the scrawl prompt of the embodiment of the application is realized.

[0116] The bus 503 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor or a local bus using any of a variety of bus structures. For example, these architectures include but are not limited to industry standard architecture (ISA) bus, micro channel architecture (MAC) bus, enhanced ISA bus, video electronics standards association (VESA) local bus and peripheral component interconnect (PCI) bus.

[0117] The electronic device 500 typically includes a variety of electronic device readable media. These media can be any available media that can be accessed by the electronic device 500, including volatile and nonvolatile media, removable and non-removable media.

[0118] The memory 501 can also include computer system readable media in the form of volatile memory, such as random access memory (RAM) 504 and / or cache memory 505. The electronic device 500 can further include other removable / non-removable, volatile / non-volatile computer system storage media. For example, a storage system 506 can be used to read / write non-removable, non-volatile magnetic media (not shown in Figure 5, commonly referred to as "hard disk drive"). Although not shown in Figure 5, a disk drive can be provided for reading / writing a removable, non-volatile magnetic disk (for example, "floppy disk") and an optical disk drive for reading / writing a removable, non-volatile optical disk (for example, CD-ROM, DVD-ROM or other optical media). In these cases, each drive can be connected to the bus 503 through one or more data media interfaces. The memory 501 can include at least one program product having a set (for example, at least one) of program modules configured to perform the functions of the embodiments of the application.

[0119] The program / utility 508 having a set (at least one) of program modules 507 can be stored in, for example, the memory 501, such program modules 507 include but are not limited to an operating system, one or more application programs, other program modules, and program data, each of which or some combination thereof can include the implementation of a network environment. The program modules 507 generally perform the functions and / or methods described in the embodiments of the application.

[0120] The electronic device 500 can also communicate with one or more external devices 509 such as a keyboard or pointing device, a display 511, etc.; other devices such as a storage device (removable and / or non-removable) including, but not limited to, magnetic or optical disks or tape and software storage, etc. The communication can occur via Input / Output (I / O) interface(s) 512. Still yet, the electronic device 500 can communicate with one or more networks, such as one or more local area networks (LANs), wide area networks (WANs), and / or the Internet, through a network adapter 513. As depicted, the network adapter 513 communicates with the other components of the electronic device 500 via bus 503. It should be appreciated that the network adapter 513 can also be connected to one or more components of the electronic device 500 in some other fashion such as over a wireless connection, etc. Other hardware and / or software modules that can be used in the electronic device 500 can also be employed without departing from the scope of the application. Such modules can include, but are not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems, etc.

[0121] The processor 502 performs various functions through running programs stored in the memory 501, such as various function applications and data processing.

[0122] It should be noted that the implementation process and technical principles of the electronic device of the present embodiment are described above in the method for processing data based on the scrawl prompt.

[0123] In order to implement the above-mentioned embodiments, the present application further provides a computer readable storage medium, and the computer readable storage medium stores computer execution instructions. When the computer execution instructions are executed by a processor, the computer execution instructions are used to implement the method provided by the above-mentioned embodiments.

[0124] In order to implement the above-mentioned embodiments, the present application further provides a computer program product, and the computer program product includes a computer program. When the computer program is executed by a processor, the computer program implements the method provided by the above-mentioned embodiments.

[0125] The collection, storage, use, processing, transmission, provision and disclosure of the user personal information involved in the present application comply with the relevant legal regulations and do not violate public order and good customs.

[0126] It is important to note that user's personal information shall be collected for legitimate and reasonable uses of the service and not shared or sold outside of those legitimate uses. Further, such collection / sharing shall occur after receiving the consent of the users, including but not limited to informing the users to read the user agreement / user notice before using the function, and signing the agreement / authorization including authorization of relevant user information. In addition, any necessary steps shall be taken to protect and secure access to such personal information data, and ensure that other individuals with access to the personal information data follow their privacy policies and procedures.

[0127] The present application contemplates that the embodiments can provide a user the ability to disable the collection or use of personal information data. That is, the present disclosure contemplates providing the user with control to permit or deny the collection of personal information data by the service or application. The user's consent is obtained before the collection of personal information data, including but not limited to, the user's agreement to the terms of service, the user's agreement to the terms of the privacy policy, and / or the user's explicit consent obtained through a menu selection or other interaction.

[0128] In the foregoing description of various embodiments described herein, reference has been made to the use of terms such as "one embodiment", "an embodiment", "some embodiments", "exemplary", "specific example", or the like. This terminology is not intended to refer to any particular embodiment or number of embodiments, but instead is intended to highlight that the embodiments described can include a variety of embodiments. It is to be understood that the use of such terms is neither intended to nor should it be taken to refer to any particular embodiment or number of embodiments. Moreover, it is to be understood that the use of the terms "first", "second", "third", etc., merely identify the names of particular features and do not necessarily imply a particular order or sequence of features. Furthermore, the use of the terms before or after, including the use of "for example", "for instance", "e.g.", "accordingly", "for the sake of simplicity", "for example only", and the like, is not intended to limit a particular feature or process or method, to the feature or process or method that follows the particular term. Also, the use of the terms "first", "second", "third", etc., merely identify the names of particular features and do not necessarily imply a particular order or sequence of features. Furthermore, the use of the terms before or after, including the use of "for example", "for instance", "e.g.", "accordingly", "for the sake of simplicity", "for example only", and the like, is not intended to limit a particular feature or process or method, to the feature or process or method that follows the particular term. In the description of the embodiments, any identification of documents, acts, items, or the like, including citations of any patents and / or patent applications, should not be construed that any such document, act, item, or the like is available as prior art to the disclosure. Any description of a document, act, item, or the like should not be construed that such document, act, item, or the like is prior art to the disclosure. Any identifying use of terms in the description of the embodiments should not be interpreted that in the prior art the term is used in the exact same way as used in the description of the embodiments or that the use of the terms in the description is the only possible way to use such terms.

[0129] Further, the terms "first", "second", "third", etc., are used herein for purposes of description and are not intended to refer to or imply a relative importance or a quantity of the referred features. That is, a feature defined with "first", "second" or "third" can include at least one of the feature. In the description of the embodiments, the meaning of "a plurality" is at least two, for example, two, three, etc., unless explicitly specifically defined otherwise.

[0130] Any process or method described in a flowchart or otherwise described herein can be understood as representing code modules, segments, or portions of code that include one or more executable instructions for accomplishing a specific logical function or operation (i.e., a state) associated with that module or portion. It should also be understood that these codes can be resident in either machine- readable medium and / or machine-readable media and executed by a machine (e.g., a computer) to perform the functions or operations specifically associated with that code. Accordingly, in the description of the embodiments above, any identification of a code in a claim should not be construed as a reference to a piece of code that is literally presented hereinafter. Rather, such identification is meant to refer to either machine-readable media, a manufacturer of one or more such codes, and / or a user of one or more such codes.

[0131] The logic and / or steps represented in the flowcharts and / or described herein, for example, can be considered as a sequence of instructions to implement logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, apparatus, or device, such as a computer-based system, processor- based system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For purposes of this specification, a "computer-readable medium" can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The computer-readable medium can be a computer- readable storage medium or a computer-readable signal medium. The computer- readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium include the following: an electrical connection having one or more wires (electrical connections), a portable computer diskette (a magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium can even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, for example, via optical scanning of the paper or other medium, then compiled, interpreted, or otherwise processed in a suitable manner, if necessary, and then stored in a computer memory.

[0132] It should be understood that aspects of the application can be implemented in hardware, software, firmware or combinations thereof. In the above embodiments, various steps or methods can be implemented in software or firmware that is stored in memory and executed by a suitable instruction execution system. As such, in some embodiments, specifically configured hardware can be used to implement at least some of the functionality described herein. For example, if implemented in hardware, the hardware can include any or a combination of the following: a discrete logic circuit having logic gates for implementing logic functions upon data signals, an application specific integrated circuit having appropriate combinational logic gates, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0133] Those of skill in the art would understand that information and signals can be represented using any of a variety of technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips that can be referenced throughout the above description can be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

[0134] In addition, each of the functional units in the various embodiments of the present application can be integrated in one processing module, or each of the units can be physically present separately, or two or more units can be integrated in one module. The integrated module can be implemented in the form of hardware or in the form of a software functional module. When the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer readable storage medium.

[0135] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it should be understood that the above embodiments are exemplary and should not be construed as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above embodiments within the scope of the present application.

Claims

1. A data processing method based on a scrawl prompt, characterized by, The method comprises the following steps: obtaining a rough prompt corresponding to a data processing task and to-be-processed data; generating a first feature vector based on embedding vectors and position vectors corresponding to respective marks in the rough prompt and the to-be-processed data; inputting the first feature vector into at least one attention residual model to perform feature conversion and enhancement, and obtaining a second feature vector; determining a task processing result corresponding to the to-be-processed data based on the second feature vector.

2. The method of claim 1, wherein, The attention residual model comprises at least one attention residual block and at least one nonlinear transformation residual block connected in sequence, and the inputting of the first feature vector into the at least one attention residual model to perform feature conversion and enhancement and obtain the second feature vector comprises: inputting the first feature vector into a first attention residual block of a first attention residual model to perform feature conversion and enhancement based on a self-attention mechanism and a residual mechanism, and obtaining an intermediate feature vector output by the first attention residual block; inputting the intermediate feature vector output by the last attention residual block into a first nonlinear transformation residual block to perform feature conversion and enhancement based on a nonlinear transformation mechanism and a residual mechanism, and obtaining an intermediate feature vector output by the first nonlinear transformation residual block; in the case where the number of attention residual models is one, taking the intermediate feature vector output by the last nonlinear transformation residual block as the second feature vector; in the case where the number of attention residual models is more than one, inputting the intermediate feature vector output by the last nonlinear transformation residual block into a next attention residual model connected in sequence, and taking the intermediate feature vector output by the last nonlinear transformation residual block in the last attention residual model as the second feature vector.

3. The method of claim 2, wherein, The inputting of the first feature vector into the first attention residual block of the first attention residual model to perform feature conversion and enhancement based on the self-attention mechanism and the residual mechanism and obtain the intermediate feature vector output by the first attention residual block comprises: inputting the first feature vector into the first attention residual block of the first attention residual model to obtain an attention weight vector corresponding to the first feature vector based on the self-attention mechanism; obtaining a third feature vector based on the attention weight vector and the first feature vector; performing residual connection on the first feature vector and the third feature vector to obtain the intermediate feature vector output by the first attention residual block.

4. The method of claim 2, wherein, The nonlinear transformation residual block comprises a feedforward neural network, and the inputting of the intermediate feature vector output by the last attention residual block into the first nonlinear transformation residual block to perform feature conversion and enhancement based on the nonlinear transformation mechanism and the residual mechanism and obtain the intermediate feature vector output by the first nonlinear transformation residual block comprises: inputting the intermediate feature vector output by the last attention residual block into the first nonlinear transformation residual block to perform nonlinear transformation on the intermediate feature vector based on the feedforward neural network and obtain a fourth feature vector; The intermediate feature vector output by the last attention residual block is connected with the fourth feature vector in residual connection to obtain an intermediate feature vector output by the first nonlinear transformation residual block.

5. The method of claim 1, wherein, The second feature vector is used to determine a task processing result corresponding to the data to be processed. The second feature vector is linearly transformed to obtain a linear transformation vector, and the linear transformation vector has the same size as a size of a vocabulary corresponding to the data processing task. The linear transformation vector is converted into a probability distribution, and the probability distribution is used to determine the task processing result.

6. The method according to any one of claims 1 to 5, characterized in that, The data processing task is a target recognition task, and the data to be processed is any one of image data, video data, and point cloud data.

7. A data processing apparatus based on a scratch hint, characterized by, The method comprises the following steps: An acquisition module is configured to acquire a data processing task corresponding to a messy prompt and data to be processed. A first processing module is configured to generate a first feature vector based on embedding vectors and position vectors corresponding to each mark in the messy prompt and the data to be processed. A second processing module is configured to input the first feature vector into at least one attention residual model to perform feature conversion and enhancement to obtain a second feature vector. A result determination module is configured to determine a task processing result corresponding to the data to be processed based on the second feature vector.

8. An electronic device, comprising: The method comprises the following steps: A processor and a memory connected with the processor in communication; The memory stores computer execution instructions; The processor executes the computer execution instructions stored in the memory to implement the method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer execution instructions, and the computer execution instructions are executed by the processor to implement the method according to any one of claims 1-6.

10. A computer program product, characterised in that, The computer program is executed by the processor to implement the method according to any one of claims 1-6.

Images