Data preprocessing method based on visual task processing, visual task processing method and task platform

Abstract

The invention provides a data preprocessing method based on visual task processing, a visual task processing method and a task platform. The method comprises the following steps: acquiring an input image of a target visual task; performing feature extraction on the input image through a preset database to obtain image features and target frame features; generating a first reference cue word based on the image feature, generating a second reference cue word based on the target frame feature, and obtaining an input reference image in a preset database based on the image feature and the image frame feature, and storing the input image, the input reference image, the first reference prompt word and the second reference prompt word as standby data for processing the target visual task. According to the method, the problems of high cost consumption of multi-source heterogeneous data and insufficient precision of a universal visual model are solved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a data preprocessing method, a vision task processing method, and a task platform based on vision task processing. Background Art

[0002] In the contemporary context of the rapid development of artificial intelligence (AI), deep learning models are the mainstream solution in computer vision. However, traditional small-scale models are limited in functionality and rely heavily on specific scene data, making them difficult to achieve effective results when processing complex tasks and recognizing open-domain samples. In contrast, general vision models built on a very large-scale pre-training paradigm demonstrate significant generalization performance advantages and the emergent nature of knowledge transfer.

[0003] However, such large models are pre-trained across modalities using large amounts of heterogeneous data, which has the following drawbacks:

[0004] 1. The collection and cleaning of multi-source heterogeneous data requires huge costs, and data distribution deviations can easily lead to incomplete knowledge representation;

[0005] 2. In high-precision scenarios such as medical image analysis and precision industrial testing, there is still a significant gap in the fine-grained reasoning capabilities of the model, and the confidence level of the output results often fails to meet application standards. Summary of the Invention

[0006] In response to the deficiencies in the prior art, the present invention provides a data preprocessing method, a visual task processing method and a task platform based on visual task processing, which solve the problems in the prior art of high cost of multi-source heterogeneous data and insufficient accuracy of general visual models.

[0007] According to an embodiment of the present invention, a first aspect provides a data preprocessing method based on visual task processing, comprising:

[0008] Get the input image of the target vision task;

[0009] Extract features from the input image using a preset database to obtain image features and target frame features;

[0010] generating a first reference prompt word based on the image features, generating a second reference prompt word based on the target frame features, obtaining an input reference image from a preset database based on the image features and the image frame features, and storing the input image, the input reference image, the first reference prompt word, and the second reference prompt word as standby data for processing the target visual task;

[0011] The preset database is associated with a plurality of visual task small models, including a plurality of task images originating from the plurality of visual task small models, as well as preset feature information of each task image and data label information of each task image.

[0012] Optionally, the image feature includes a first image feature obtained by searching based on the preset feature information, and a second image feature obtained by matching semantic features based on the data tag information;

[0013] The target frame feature is retrieved based on the data label information.

[0014] Optionally, performing a search based on the preset feature information to obtain the first image feature includes:

[0015] Processing the input image through an image encoder to obtain input image features;

[0016] The input image feature is matched with a plurality of preset feature information to obtain a first image feature.

[0017] Optionally, performing semantic feature matching based on the data tag information to obtain a second image feature includes:

[0018] Processing the input image through an image encoder to obtain input image features;

[0019] For each task image, the data label information is output to a large language model to obtain a first text description, and the first text description is processed by a text encoder to obtain semantic features; the first text description includes at least the category of the target object, the number of the target objects, and the location of the target objects;

[0020] The input image feature is matched with multiple semantic features to obtain a second image feature.

[0021] Optionally, searching based on the data tag information to obtain target frame features includes:

[0022] Inputting the input image into the preset database, and predicting the input image based on the data label information to obtain predicted label information;

[0023] For each task image, cropping the task image according to the predicted label information to obtain a preset image frame image; the predicted label information includes at least the category of the image frame, the number of the image frames, and the position of the image frames;

[0024] Cropping the input image according to the predicted label information to obtain an image frame image to be matched;

[0025] Processing the preset image frame image through an image encoder to obtain a plurality of preset image frame features, and processing the image frame image to be matched through an image encoder to obtain image frame features to be matched;

[0026] The to-be-matched image frame features are similarly matched with a plurality of preset image frame features to obtain target frame features.

[0027] Optionally, obtaining an input reference image in the preset database based on the image feature and the image frame feature includes:

[0028] According to the image features and the target frame features, K task images with similarities greater than a preset value are obtained in the preset database and stored as input reference images.

[0029] Optionally, after obtaining the input image of the target visual task, the method further includes:

[0030] Performing feature extraction on the input image to obtain image features;

[0031] A second reference prompt word is generated based on the image feature, an input reference image is obtained from the preset database based on the image feature, and the input image, the input reference image, and the second reference prompt word are stored as stand-by data for processing the target visual task.

[0032] Optionally, obtaining an input reference image from the preset database based on the image feature includes:

[0033] According to the image features, K task images with similarities greater than a preset value are obtained from the preset database and stored as input reference images.

[0034] A second aspect provides a visual task processing method, comprising:

[0035] Obtaining standby data for processing a target visual task according to the data preprocessing method based on visual task processing as described above;

[0036] Input the unused data into the LLM module;

[0037] The output of the LLM module is processed by a decoder set to obtain a target processing result of the target visual task, and the decoder set includes at least a classification decoder, a detection decoder, and a segmentation decoder.

[0038] Optionally, the decoder set further includes a text decoder;

[0039] The text decoder is used to process the target processing result to generate an image description.

[0040] A third aspect provides a task platform, including a request interface, a visual processing interface, and a response unit;

[0041] The request interface is used to receive a processing request for processing a target visual task sent by a terminal device, wherein the processing request includes an input image and an image processing type;

[0042] The response unit is configured to, based on the processing request, schedule a data pre-processing device based on visual task processing through the visual processing interface to process the input image to obtain standby data for processing the target visual task; and schedule a visual task processing device through the visual processing interface to process the standby data for the target visual task to obtain a target processing result for the target visual task;

[0043] The data preprocessing device based on visual task processing implements the data preprocessing method based on visual task processing as described above; the visual task processing device implements the visual task processing method as described above.

[0044] The fourth aspect provides an electronic device, including a memory, a processor, and a computer program stored in the memory and runnable on the processor, wherein when the processor executes the computer program, the data preprocessing method as described above, or the visual task processing method as described above, or the task platform as described above is implemented.

[0045] The fifth aspect provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data preprocessing method based on visual tasks as described above, or the visual task processing method as described above, or the task platform as described above.

[0046] The sixth aspect provides a computer program product, including a computer program, which, when executed by a processor, implements the data preprocessing method based on visual tasks as described above, or the visual task processing method as described above, or the task platform as described above.

[0047] The technical principle of the present invention is: data preprocessing is performed on the input image of the target visual task in conjunction with a preset database to obtain an input reference image, a first reference prompt word, and a second reference prompt word that are of reference significance for visual task processing, and the input image, input reference image, first reference prompt word, and second reference prompt word are stored together as stand-by data for the target visual task.

[0048] Compared with the existing technology, the present invention has the following beneficial effects: by introducing the existing business system through the preset database, the accuracy and reliability of visual task processing can be improved without the need to rebuild new scene data. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 Schematic diagram of the implementation flow of the data preprocessing method based on visual task processing according to an embodiment of the present invention;

[0050] Figure 2 A schematic diagram of the process of building a preset database according to an embodiment of the present invention;

[0051] Figure 3 Schematic diagram of the implementation flow of another data preprocessing method based on visual task processing according to an embodiment of the present invention;

[0052] Figure 4 Schematic diagram of the implementation flow of the visual task processing method according to an embodiment of the present invention;

[0053] Figure 5 This is a schematic diagram of the structure of the task platform according to an embodiment of the present invention;

[0054] Figure 6 Schematic diagram of the structure of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0055] The technical solutions of the present invention are further described below with reference to the accompanying drawings and embodiments.

[0056] like Figure 1 As shown, the embodiment of the present invention proposes a data preprocessing method based on visual task processing, including but not limited to the following steps:

[0057] Sa1, obtain the input image of the target visual task;

[0058] Sa21, extracting features of the input image through a preset database to obtain image features and target frame features;

[0059] Sa31, generate a first reference prompt word based on the image features, generate a second reference prompt word based on the target frame features, obtain an input reference image in the preset database based on the image features and the image frame features, and store the input image, the input reference image, the first reference prompt word, and the second reference prompt word as stand-by data for processing the target visual task.

[0060] In the above step Sa21, the preset database is a pre-established database associated with multiple visual task small models, which stores the training data of all visual task small models. The multiple visual task small models can come from other business systems based on visual tasks, such as an autonomous driving system, which includes an obstacle detection small model based on image classification, a security monitoring small model based on target detection, a target positioning small model based on target detection, etc.; or a medical image analysis system, which includes a category labeling small model based on semantic segmentation, a lesion detection small model based on image classification, a cell counting small model based on instance segmentation, etc.; or an intelligent industrial system, which includes a defect classification small model based on image classification, an industrial parts assembly detection small model based on key point detection, a product statistics small model based on instance segmentation, etc.

[0061] It is understandable that the training data for the visual task mini-models is data that has been classified and annotated according to its visual task, such as the aforementioned obstacle detection, security monitoring, target positioning, category labeling, lesion detection, etc. Therefore, in an embodiment of the present invention, the preset database is associated with multiple visual task mini-models, including multiple task images derived from multiple visual task mini-models, as well as preset feature information for each task image and data label information for each task image, wherein the preset feature information represents the image features in the task image used to solve the visual task, and the data label information represents a text description of the image features in the task image used to solve the visual task, including category, quantity, location, etc.

[0062] Exemplarily, for multiple visual task small models from an autonomous driving system, the preset database stores task image A of the obstacle detection small model, task image B of the security monitoring small model, and task image C of the target positioning small model; based on this, the preset feature information stored in the preset database is: obstacle feature information of each task image A, dangerous target feature information of each task image B, and target position feature information of each task image C; based on this, the data label information stored in the preset database is: category, quantity, and position of obstacle feature information of each task image A, category, quantity, and position of dangerous target feature information of each task image B, and category, quantity, and position of target position feature information of each task image C.

[0063] In order to ensure that the task image with the highest similarity to the input image in the preset database is found as the input reference image, similarity comparison is required from multiple aspects. The embodiment of the present invention provides a hybrid retrieval method based on image feature retrieval and semantic feature retrieval. Therefore, the image feature of the above step Sa21 includes a first image feature obtained by retrieval based on the preset feature information, and a second image feature obtained by semantic feature matching based on the data tag information; the target box feature of the above step Sa21 is retrieved based on the data tag information.

[0064] In one embodiment, searching and obtaining the first image feature based on the preset feature information includes:

[0065] Sa211, processing the input image through an image encoder to obtain input image features;

[0066] Sa212: Perform similarity matching on the input image feature and multiple preset feature information to obtain a first image feature.

[0067] It should be noted that, in a preferred implementation, the first image feature of Sa212 is preset feature information in a preset database that has the highest similarity to the input image feature.

[0068] In one embodiment, performing semantic feature matching based on the data tag information to obtain the second image feature includes:

[0069] Sa213, processing the input image through an image encoder to obtain input image features;

[0070] Sa214: For each task image, output the data label information to the large language model to obtain a first text description, and process the first text description through a text encoder to obtain semantic features; the first text description includes at least the category of the target object, the number of target objects, and the location of the target object;

[0071] Sa215: Perform similarity matching on the input image feature and multiple semantic features to obtain a second image feature.

[0072] It should be noted that the target object in step Sa214 is a specific object in the image that needs to be identified, segmented, or tracked, such as a face, vehicle, or text. In a preferred implementation, the second image feature in Sa214 includes data label information in a preset database that has the highest semantic similarity to the input image feature.

[0073] In one embodiment, searching and obtaining target frame features based on the data tag information includes:

[0074] Sa216, inputting the input image into the preset database, and predicting the input image based on the data label information to obtain predicted label information;

[0075] Sa217. For each task image, crop the task image according to the predicted label information to obtain a preset image frame image; the predicted label information includes at least the category of the image frame, the number of the image frames, and the position of the image frames;

[0076] Sa218, cropping the input image according to the predicted label information to obtain an image frame image to be matched;

[0077] Sa219, processing the preset image frame image through an image encoder to obtain a plurality of preset image frame features, and processing the to-be-matched image frame image through an image encoder to obtain to-be-matched image frame features;

[0078] Sa2110: Perform similarity matching on the image frame feature to be matched with multiple preset image frame features to obtain target frame features.

[0079] Regarding the above step Sa216, the embodiment of the present invention provides a detailed description of the construction of the preset database. Figure 2 As shown, the embodiment of the present invention first stores data in a training data part 101 and an inference data part 102 respectively. Among them, the training data part 101 is the training data of the visual task small model. Since the visual task small model is an existing model, its training data includes task images and annotation information. According to the foregoing, the preset database of the embodiment of the present invention includes multiple task images derived from multiple visual task small models, as well as preset feature information of each task image and data label information of each task image. When establishing a preset database based on the training data, the task images of the training data are directly stored, and the preset feature information is stored after the task images are extracted by the image encoder 103, and the annotation information of the task images is directly stored as the data label information of the task images. When new data, such as an image, is input into the preset database 100, the preset database 100 stores the new data, i.e., the input image, in the inference data portion 102. At this time, the new data (input image) has no label information. Therefore, as described above, Sa211 and Sa213, the input image features must be obtained through the image encoder 103. The label information must also be predicted by the corresponding visual task mini-model 104 to obtain the predicted label information, which is then placed in the preset database for processing such as Sa212, Sa215, and Sa217 to Sa2110. For example, when the input image belongs to the business domain of the autonomous driving system, multiple visual task mini-models of the autonomous driving system are called to perform the above predictions.

[0080] It should be noted that the target frame feature in the above Sa2110 is a preset image frame in a preset database that has the highest similarity with the image frame to be matched.

[0081] In step Sa31 above, the first and second reference prompts consist of both image features and text. The text portion is normalized by a text encoder before use, while the image features are used directly. Furthermore, it should be noted that the input reference images include K images, where K is a positive integer greater than or equal to 1. Accordingly, the first and second reference prompts should each contain K examples.

[0082] In the embodiment of the present invention, the first reference prompt word generated based on the image features and the second reference prompt word generated based on the target frame features in the above step Sa31 are described using a practical example. First, assuming that two input reference images are obtained based on the above image features and target frame features, the first reference prompt word and the second reference prompt word include:

[0083] Example 1:

[0084] Input: {image features}

[0085] Output: {(image category description), (object box (x1, y1, x2, y2)), (object mask)}; {…}

[0086] Example 2:

[0087] Input: {image features}

[0088] Output: {{object 1 category}, {object 2 category}, ...}, {{object 1 box (x1, y1, x2, y2)}, {object 2 box}, ...}, {object 1 mask features, object 2 mask features, ...}

[0089] Now, please complete accurate recognition, detection, and segmentation of {image features}.

[0090] Here, {image feature} represents the first reference prompt, {(image category description), (object bounding box (x1, y1, x2, y2)), (object mask)}; and {…} represents the second reference prompt. It is understood that the number of examples can be determined based on the number of similar images retrieved from the database. While the task type can be adjusted based on the specific situation, the three most common examples are used here.

[0091] The embodiment of the present invention uses image features and image frame features according to actual application scenarios. Specifically, step Sa31 is applicable to visual tasks with clear target frames, such as target detection and segmentation, and therefore uses both image features and image frame features. Therefore, step Sa31 obtains an input reference image from the preset database based on the image features and the image frame features, including:

[0092] Sa311. Obtain K task images with a similarity greater than a preset value from the preset database based on the image features and the target frame features, and store them as input reference images.

[0093] In another embodiment, for vision tasks that do not require object detection and segmentation, only image features are used. Figure 3 As shown, in Figure 1 The data preprocessing method based on visual task processing shown above also includes but is not limited to the following steps:

[0094] Sa22, performing feature extraction on the input image to obtain image features;

[0095] Sa32. Generate a second reference prompt word based on the image features, obtain an input reference image in the preset database based on the image features, and store the input image, the input reference image, and the second reference prompt word as stand-by data for processing the target visual task.

[0096] Then, in the above-mentioned Sa32, obtaining an input reference image from the preset database based on the image features includes:

[0097] Sa321. Obtain K task images with similarity greater than a preset value from the preset database according to the image features, and store them as input reference images.

[0098] like Figure 4 As shown, in another embodiment of the present invention, a visual task processing method is further provided, comprising:

[0099] Sb1. Obtaining standby data for processing a target visual task according to the above-mentioned data preprocessing method based on visual task processing;

[0100] Sb2, inputting the unused data into the LLM module;

[0101] Sb3. Process the output of the LLM module through a decoder set to obtain the target processing result of the target visual task, where the decoder set includes at least a classification decoder, a detection decoder, and a segmentation decoder.

[0102] According to the above steps Sb1 to Sb3, the embodiment of the present invention integrates the business system built based on a specific scenario into the large model. Therefore, some parameters of the LLM (Large Language Model) module are also fine-tuned to adapt it to image feature embedding. In practical applications, when fine-tuning the model, the fine-tuning parts can be: some LLM parameters, classification decoder, detection decoder, segmentation decoder; the fine-tuning data can be all the training data of the original CV system or new unlabeled data. If unlabeled data is used, the labeling information of the new image can come from the CV model of the original system. The training loss function is consistent with the CV model training loss function, such as: classification uses cross entropy loss, segmentation model uses cross entropy loss, Dice loss, etc., and target detection model uses L1, L2 loss, etc.

[0103] In one embodiment, the decoder set further comprises a text decoder;

[0104] The text decoder is used to process the target processing result to generate an image description. Based on this, in addition to outputting accurate classification, detection, and segmentation results, the embodiment of the present invention can also output the description as a piece of text for easy use.

[0105] like Figure 5 As shown, in another embodiment of the present invention, a task platform 500 is further provided, comprising a request interface 501, a visual processing interface 502, and a response unit 503;

[0106] The request interface 501 is used to receive a processing request for processing a target visual task sent by a terminal device, wherein the processing request includes an input image and an image processing type;

[0107] The response unit 503 is configured to, based on the processing request, schedule a data pre-processing device based on visual task processing through the visual processing interface 502 to process the input image to obtain standby data for processing the target visual task; and schedule a visual task processing device through the visual processing interface 502 to process the standby data for the target visual task to obtain a target processing result for the target visual task;

[0108] The data preprocessing device based on visual task processing implements the above-mentioned data preprocessing method based on visual task processing; the visual task processing device implements the above-mentioned visual task processing method.

[0109] like Figure 6, which is a schematic diagram of the structure of an electronic device 60 that can be used to implement an embodiment of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0110] like Figure 6 As shown, the electronic device 60 includes at least one processor 61 and a memory connected to the at least one processor 61, such as a read-only memory (ROM) 62, a random access memory (RAM) 63, etc., wherein the memory stores a computer program that can be executed by the at least one processor, and the processor 61 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 62 or the computer program loaded from the storage unit 68 to the random access memory (RAM) 63. Various programs and data required for the operation of the electronic device 60 can also be stored in the RAM 63. The processor 61, ROM 62 and RAM 63 are connected to each other via a bus 64. The input / output (I / O) interface AA is also connected to the bus 64.

[0111] Multiple components in the electronic device 60 are connected to the I / O interface AA, including an input unit 66, such as a keyboard, a mouse, etc.; an output unit 67, such as various types of displays, speakers, etc.; a storage unit 68, such as a magnetic disk, an optical disk, etc.; and a communication unit 69, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 69 allows the electronic device 60 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0112] Processor 61 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of processor 61 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any other suitable processors, controllers, microcontrollers, etc. Processor 61 executes the various methods and processes described above, such as the optimization method based on the gun-ball linkage monitoring system.

[0113] In some embodiments, the data processing method can be implemented as a computer program that is tangibly contained in a computer-readable storage medium, such as a storage unit 68. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 60 via the ROM 62 and / or the communication unit 69. When the computer program is loaded into the RAM 63 and executed by the processor 61, one or more steps of the data processing method described above can be performed. Alternatively, in other embodiments, the processor 61 can be configured to perform the data processing method in any other suitable manner (e.g., by means of firmware).

[0114] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0115] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the computer program is executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program may be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0116] In the context of the present invention, computer-readable storage media can be tangible media that can contain or store a computer program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Computer-readable storage media can include but are not limited to electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, computer-readable storage media can be machine-readable signal media. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0117] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0118] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0119] A computing system may include clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services.

[0120] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved. This is not limited herein.

[0121] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A data preprocessing method based on visual task processing, characterized in that: include: Get the input image of the target vision task; Extract features from the input image using a preset database to obtain image features and target frame features; generating a first reference prompt word based on the image features, generating a second reference prompt word based on the target frame features, obtaining an input reference image from a preset database based on the image features and the image frame features, and storing the input image, the input reference image, the first reference prompt word, and the second reference prompt word as standby data for processing the target visual task; The preset database is associated with a plurality of visual task small models, including a plurality of task images originating from the plurality of visual task small models, as well as preset feature information of each task image and data label information of each task image.

2. The data preprocessing method based on visual task processing according to claim 1, characterized in that: The image features include a first image feature obtained by searching based on the preset feature information, and a second image feature obtained by semantic feature matching based on the data tag information; The target frame feature is retrieved based on the data label information.

3. The data preprocessing method based on visual task processing according to claim 2, characterized in that: Retrieving and obtaining a first image feature based on the preset feature information includes: Processing the input image through an image encoder to obtain input image features; The input image feature is matched with a plurality of preset feature information to obtain a first image feature.

4. The data preprocessing method based on visual task processing according to claim 2, characterized in that: Performing semantic feature matching based on the data tag information to obtain a second image feature includes: Processing the input image through an image encoder to obtain input image features; For each task image, the data label information is output to a large language model to obtain a first text description, and the first text description is processed by a text encoder to obtain semantic features; the first text description includes at least the category of the target object, the number of the target objects, and the location of the target objects; The input image feature is matched with multiple semantic features to obtain a second image feature.

5. The data preprocessing method based on visual task processing according to claim 2, characterized in that: Retrieving the target frame features based on the data tag information includes: Inputting the input image into the preset database, and predicting the input image based on the data label information to obtain predicted label information; For each task image, cropping the task image according to the predicted label information to obtain a preset image frame image; the predicted label information includes at least the category of the image frame, the number of the image frames, and the position of the image frames; Cropping the input image according to the predicted label information to obtain an image frame image to be matched; Processing the preset image frame image through an image encoder to obtain a plurality of preset image frame features, and processing the image frame image to be matched through an image encoder to obtain image frame features to be matched; The to-be-matched image frame features are similarly matched with a plurality of preset image frame features to obtain target frame features.

6. The data preprocessing method based on visual task processing according to any one of claims 1 to 5, characterized in that: Obtaining an input reference image in the preset database based on the image feature and the image frame feature includes: According to the image features and the target frame features, K task images with similarities greater than a preset value are obtained in the preset database and stored as input reference images.

7. The data preprocessing method based on visual task processing according to any one of claims 1 to 5, characterized in that: After obtaining the input image of the target visual task, the method further includes: Performing feature extraction on the input image to obtain image features; A second reference prompt word is generated based on the image feature, an input reference image is obtained from the preset database based on the image feature, and the input image, the input reference image, and the second reference prompt word are stored as stand-by data for processing the target visual task.

8. The data preprocessing method based on visual task processing according to claim 7, characterized in that: Obtaining an input reference image in the preset database based on the image feature includes: According to the image features, K task images with similarities greater than a preset value are obtained from the preset database and stored as input reference images.

9. A visual task processing method, characterized in that: include: Obtaining standby data for processing a target visual task according to the data preprocessing method based on visual task processing according to any one of claims 1 to 8; Input the unused data into the LLM module; The output of the LLM module is processed by a decoder set to obtain a target processing result of the target visual task, and the decoder set includes at least a classification decoder, a detection decoder, and a segmentation decoder.

10. The visual task processing method according to claim 9, wherein: The decoder set also includes a text decoder; The text decoder is used to process the target processing result to generate an image description.

11. A task platform, comprising a request interface, a visual processing interface, and a response unit; The request interface is used to receive a processing request for processing a target visual task sent by a terminal device, wherein: The processing request includes an input image and an image processing type; The response unit is configured to schedule, based on the processing request, a data pre-processing device based on visual task processing through the visual processing interface to process the input image to obtain standby data for processing the target visual task; and scheduling a visual task processing device through the visual processing interface to process the standby data of the target visual task to obtain a target processing result of the target visual task; The data preprocessing device based on visual task processing implements the data preprocessing method based on visual task processing according to any one of claims 1 to 8; the visual task processing device implements the visual task processing method according to claim 9 or 10.

12. An electronic device, characterized in that: It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements the data preprocessing method according to any one of claims 1 to 8, or the visual task processing method according to any one of claims 9 to 10, or the task platform according to claim 11.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, it implements the data preprocessing method based on a visual task as described in any one of claims 1 to 8, or the visual task processing method as described in any one of claims 9 to 10, or the task platform as described in claim 11.

14. A computer program product comprising a computer program, characterized in that When executed by a processor, the computer program implements the data preprocessing method based on a visual task as described in any one of claims 1 to 8, or the visual task processing method as described in any one of claims 9 to 10, or the task platform as described in claim 11.

Images