Image processing method and device, computer equipment and readable storage medium

Abstract

The invention relates to an image processing method and device, computer equipment and a readable storage medium. The method comprises the following steps: inputting a target image into a target detection model and a multi-modal model to obtain size information and a global description text of the target image as well as category information, confidence, coordinate information and an attribute description text of a target, and only sending the obtained information to a cloud, so as to obtain a target detection result; therefore, the cloud can restore the target image based on the information without sending the complete target image, so that the network delay can be effectively reduced, and the transmission efficiency is improved.

Description

Technical Field

[0001] This application relates to the fields of cloud computing and large model inference technology, and in particular to an image processing method, apparatus, computer device and readable storage medium. Background Technology

[0002] With the rapid development of artificial intelligence and Internet of Things technologies, how to achieve efficient, low-latency visual perception and intelligent decision-making has become an important issue in fields such as industrial inspection, autonomous driving, and intelligent security.

[0003] Currently, the main approach is to directly upload the raw images to the cloud for large-scale model analysis. However, as application scenarios increasingly demand real-time performance, computational efficiency, and semantic understanding depth, the method of transmitting raw images relies on high-bandwidth transmission. In particular, with the increasing number of terminals, higher resolutions, and larger image data, network latency becomes high and transmission efficiency decreases. Summary of the Invention

[0004] Therefore, it is necessary to provide an image processing method, apparatus, computer device, and readable storage medium that can improve transmission efficiency in response to the above-mentioned technical problems.

[0005] In a first aspect, this application provides an image processing method applied to edge nodes, the method comprising:

[0006] Receive the target image sent by the image acquisition device, and obtain the size information of the target image;

[0007] The target image is input into the target detection model, which outputs the category information, confidence score, and coordinate information of any target in the target image.

[0008] The target image is input into a multimodal model, which outputs the global description text of the target image, the first text length of the global description text, the attribute description text of any target, and the second text length of the attribute description text.

[0009] Based on the first text length, the second text length, and the category information, the size information, the category information, the confidence level, the coordinate information, the global description text, and the attribute description text are sent to the cloud.

[0010] In one embodiment, sending the size information, the category information, the confidence level, the coordinate information, the global description text, and the attribute description text to the cloud based on the first text length, the second text length, and the category information includes:

[0011] The size information, category information, confidence level, coordinate information, global description text, and attribute description text are converted into structured data.

[0012] According to the first preset encoding rule, the first number of bits of the structured data is obtained based on the first text length, the second text length, and the category information;

[0013] Based on the first number of bits, the structured data is sent to the cloud.

[0014] In one embodiment, sending the structured data to the cloud based on the first number of bits includes:

[0015] Based on the first number of bits, obtain the first data size of the structured data, and based on the first data size, obtain the first transmission delay required to transmit the structured data to the cloud;

[0016] Obtain the second transmission delay required to transmit the image data of the target image to the cloud;

[0017] If the first transmission delay is greater than the second transmission delay, the image data will be sent to the cloud.

[0018] If the first transmission delay is not greater than the second transmission delay, the structured data is sent to the cloud.

[0019] In one embodiment, obtaining the second transmission delay required to transmit the image data of the target image to the cloud includes:

[0020] Obtain the target image's size information, compression ratio, number of color channels, and number of bits per color channel;

[0021] According to the second preset encoding rule, the second number of bits of the image data is obtained based on the size information, the compression ratio, the number of channels, and the number of channel bits, and the second transmission delay is obtained based on the second number of bits.

[0022] In one embodiment, the multimodal model includes a fixed question template that restricts the output rules for the descriptive text.

[0023] Secondly, this application also provides an image processing method applied in the cloud, the method comprising:

[0024] Receive the size information and global description text of the target image sent by the edge node, as well as the category information, confidence level, coordinate information and attribute description text of any target in the target image;

[0025] For any given target, the target's category information, confidence level, coordinate information, and attribute description text are embedded into a fixed prompt word template;

[0026] Sort the fixed prompt word templates for all targets in descending order of confidence level;

[0027] The target image's size information, global description text, and sorted fixed prompt word template are input into the text-based image model, and the target image is output.

[0028] Thirdly, this application also provides an image processing apparatus, the apparatus comprising:

[0029] The first receiving module is used to receive the target image sent by the image acquisition device and obtain the size information of the target image;

[0030] The first input module is used to input the target image into the target detection model and output the category information, confidence score and coordinate information of any target in the target image;

[0031] The second input module is used to input the target image into the multimodal model and output the global description text of the target image, the first text length of the global description text, the attribute description text of any target, and the second text length of the attribute description text.

[0032] The sending module is used to send the size information, the category information, the confidence level, the coordinate information, the global description text, and the attribute description text to the cloud based on the first text length, the second text length, and the category information.

[0033] Fourthly, this application also provides an image processing apparatus, the apparatus comprising:

[0034] The second receiving module is used to receive the size information and global description text of the target image sent by the edge node, as well as the category information, confidence level, coordinate information and attribute description text of any target in the target image;

[0035] An embedding module is used to embed the category information, confidence level, coordinate information, and attribute description text of any target into a fixed prompt word template.

[0036] The sorting module is used to sort the fixed prompt word templates of all targets in descending order of confidence level;

[0037] The third input module is used to input the size information of the target image, the global description text, and the sorted fixed prompt word template into the text-to-image model, and output the target image.

[0038] Fifthly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the methods in any of the above embodiments.

[0039] Sixthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the steps of the methods in any of the above embodiments.

[0040] The aforementioned image processing method, apparatus, computer equipment, and readable storage medium receive a target image sent by an image acquisition device and obtain the size information of the target image; input the target image into a target detection model and output the category information, confidence score, and coordinate information of any target in the target image; input the target image into a multimodal model and output the global description text of the target image, the first text length of the global description text, the attribute description text of any target, and the second text length of the attribute description text; based on the first text length, the second text length, and the category information, send the size information, category information, confidence score, coordinate information, global description text, and attribute description text to the cloud. The method provided in this application, by inputting the target image into a target detection model and a multimodal model to obtain the size information and global description text of the target image, as well as the category information, confidence score, coordinate information, and attribute description text of the target, only needs to send this information to the cloud, allowing the cloud to reconstruct the target image based on this information, without needing to send the complete target image, effectively reducing network latency and improving transmission efficiency. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is an application environment diagram of an image processing method in one embodiment;

[0043] Figure 2 This is a flowchart illustrating an image processing method in one embodiment;

[0044] Figure 3This is a flowchart illustrating a structured data transmission method in one embodiment;

[0045] Figure 4 This is a flowchart illustrating the image processing method in the second embodiment;

[0046] Figure 5 This is a structural block diagram of an image processing device in one embodiment;

[0047] Figure 6 This is a structural block diagram of the image processing device in the second embodiment;

[0048] Figure 7 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0050] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0051] The image processing method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, the image acquisition device 101 can be, but is not limited to, a camera or sensor; the edge node 102 is a computing device or node close to the image acquisition device 101, typically deployed at the end or edge of the physical world; and the cloud 103 is a remote, large-scale data center providing powerful distributed computing and storage capabilities, offering services via the Internet. Specifically, the image acquisition device 101 sends a target image to the edge node 102, the edge node 102 extracts key information from the target image, and sends the key information to the cloud 103, which then reconstructs the target image based on the key information.

[0052] In one exemplary embodiment, such as Figure 2 As shown, an image processing method is provided, which is applied to... Figure 1 Taking an edge node as an example, the explanation includes steps 202 to 208. Wherein:

[0053] S202. Receive the target image sent by the image acquisition device and obtain the size information of the target image.

[0054] Optionally, image size information can be used In this representation, H represents the height of the target image, W represents the width of the target image, and C represents the number of channels of a pixel in the target image. Since RGB images typically have 3 channels, the target image can be represented as... .

[0055] S204. Input the target image into the target detection model and output the category information, confidence score, and coordinate information of any target in the target image.

[0056] Among them, category information represents which category of preset objects the target belongs to. For example, preset objects may include people, dogs, cars, and trees; confidence level represents the degree to which the detected area is a real target.

[0057] Optionally, the object detection model may be, but is not limited to, the YOLO (You Only Look Once) model, which performs both category identification and location localization of objects in an image through a single forward propagation of a neural network.

[0058] Alternatively, the set of all targets in the target image can be used Let i represent each target, and there are N targets in total. , Information indicating the category of the target. Indicates the confidence level of the target. This represents the coordinates of the center point of the target detection box. This represents the coordinates of the top-left corner of the target detection box.

[0059] S206. Input the target image into the multimodal model and output the global description text of the target image, the first text length of the global description text, the attribute description text of any target, and the second text length of the attribute description text.

[0060] Optionally, a multimodal model refers to an artificial intelligence model that can simultaneously process and fuse two different data modalities, namely images and text, to achieve cross-modal understanding and information generation. A multimodal model can be, but is not limited to, ViLBERT (Vision-and-Language BERT, a bidirectional encoder representation model for vision and language). ViLBERT is a deep learning model that focuses on cross-modal fusion of vision and text. It is an extension of the classic text model BERT architecture. Its core goal is to enable AI to understand image information and text information at the same time and establish semantic relationships between the two.

[0061] Optionally, by inputting the target image into a multimodal model, key semantic information of the target image can be extracted. This key semantic information includes global descriptive text describing the target image as a whole and attribute descriptive text describing each individual target. The attribute descriptive text of the target can be represented by a set of attributes. express, It may include, but is not limited to, at least one of the following attributes: color, action, relationship, etc.

[0062] Optionally, the text length may be represented, but is not limited to, the number of tokens.

[0063] Optionally, the multimodal model includes a fixed question template that restricts the output rules of the descriptive text. The fixed question template can be used to limit the output range, format and content focus of the descriptive text, thus avoiding the generation of redundant text. For example, the global descriptive text of a target image can be "a person is walking a dog on the road", and the attribute descriptive text of a certain target can be "a yellow dog is running".

[0064] S208. Based on the first text length, the second text length, and the category information, send the size information, category information, confidence level, coordinate information, global description text, and attribute description text to the cloud.

[0065] Optionally, the size of the data to be sent to the cloud can be obtained first based on the first text length, the second text length, and the category information. Then, the data transmission latency can be predicted based on the data size, and the data transmission latency can be used to determine whether to send the data to the cloud.

[0066] In the aforementioned image processing method, a target image sent by an image acquisition device is received, and its size information is obtained. The target image is then input into a target detection model, which outputs the category information, confidence score, and coordinate information of any target in the target image. The target image is then input into a multimodal model, which outputs the global description text of the target image, the first text length of the global description text, the attribute description text of any target, and the second text length of the attribute description text. Based on the first text length, the second text length, and the category information, the size information, category information, confidence score, coordinate information, global description text, and attribute description text are sent to the cloud. The method provided in this application, by inputting the target image into a target detection model and a multimodal model, obtains the target image's size information, global description text, and the target's category information, confidence score, coordinate information, and attribute description text. Only this information needs to be sent to the cloud, allowing the cloud to reconstruct the target image based on this information, without needing to send the complete target image. This effectively reduces network latency and improves transmission efficiency.

[0067] In some embodiments, such as Figure 3As shown, based on the first text length, the second text length, and category information, size information, category information, confidence level, coordinate information, global description text, and attribute description text are sent to the cloud, including:

[0068] S302. Convert the size information, category information, confidence level, coordinate information, global description text, and attribute description text into structured data.

[0069] S304. According to the first preset encoding rule, obtain the first number of bits of the structured data based on the first text length, the second text length and the category information.

[0070] S306. Based on the first number of bits, send the structured data to the cloud.

[0071] Bits are the smallest unit of measurement for data volume in a computer and can be used to measure file size.

[0072] Optionally, the target detection information and semantic description information can be converted into a structured data packet. The structured data packet includes image metadata (height, width, channels), target information (category, confidence, center coordinates, top-left coordinates), and semantic description (global description, description of each target). The structured data packet can be in JSON format.

[0073] Optionally, the first preset encoding rule can be as follows:

[0074]

[0075] In the formula, The number of bits for structured data. The number of bits for encoding a single numerical value. This indicates the category information for each target. The second text length represents the attribute description text for each target. Indicates the first text length of the global description text.

[0076] In this embodiment, by sending structured data to the cloud based on the first number of bits, more accurate data transmission is possible.

[0077] In some embodiments, sending structured data to the cloud based on a first number of bits includes: obtaining a first data size of the structured data based on the first number of bits, and obtaining a first transmission delay required to transmit the structured data to the cloud based on the first data size; obtaining a second transmission delay required to transmit the image data of the target image to the cloud; sending the image data to the cloud if the first transmission delay is greater than the second transmission delay; and sending the structured data to the cloud if the first transmission delay is not greater than the second transmission delay.

[0078] Image data refers to the set of all binary information used to digitally represent an image, and is the underlying carrier that enables images to be stored, transmitted, processed, and displayed.

[0079] Optionally, the first transmission delay can be calculated by the following formula:

[0080]

[0081] In the formula, T is the first transmission delay, B is the communication bandwidth (bits / s), and D is the network delay constant.

[0082] In this embodiment, by comparing the first transmission delay and the second transmission delay, it is determined whether to send structured data or image data of the target image. This ensures that the target image data is sent to the cloud while also guaranteeing the real-time performance and reliability of data transmission.

[0083] In some embodiments, obtaining the second transmission delay required to transmit image data to the cloud includes: obtaining the size information, compression ratio, number of color channels, and number of channel bits for any color channel of the target image; obtaining the second number of bits of the image data based on the size information, compression ratio, number of channels, and number of channel bits according to a second preset encoding rule, and obtaining the second transmission delay based on the second number of bits.

[0084] Optionally, the second transmission delay can be calculated using the following formula:

[0085]

[0086] In the formula, This is the second number of bits.

[0087] Optionally, the second preset encoding rule can be as follows:

[0088]

[0089] In the formula, H is the length of the target image (in pixels), W is the width of the target image (in pixels), and C is the number of color channels in the target image. R represents the number of channel bits for a single color channel, and R represents the compression ratio of the target image. The compression ratio of a JPEG image is typically 90%.

[0090] In this embodiment, the second transmission delay required to transmit image data to the cloud is determined based on the second preset encoding rule, making the determined second transmission delay more accurate and laying the foundation for the subsequent process of determining the data transmitted to the cloud.

[0091] In one exemplary embodiment, such as Figure 4As shown, a second image processing method is provided, which is applied to... Figure 1 Taking the cloud as an example, the explanation includes steps 402 to 408. Among them:

[0092] S402, Receive the size information and global description text of the target image sent by the edge node, as well as the category information, confidence level, coordinate information and attribute description text of any target in the target image.

[0093] Optionally, the cloud receives a structured data packet sent by the edge node. The structured data packet includes the size information and global description text of the target image, as well as the category information, confidence level, coordinate information, and attribute description text of any target in the target image.

[0094] Optionally, after receiving data sent by the edge node, the attribute description text of any target can be aligned with the category information, confidence level, and coordinate information.

[0095] S404. For any target, embed the target's category information, confidence level, coordinate information, and attribute description text into a fixed prompt word template.

[0096] Optionally, a fixed prompt template is a pre-designed text framework used to standardize the organization and expression of information. Its core function is to transform the structured data (category, coordinates, description, etc.) obtained from target detection into fill-in-the-blank text instructions that large models can understand. For example, a fixed prompt template for a target could be "There is a [category (human)] in the image, located at [center (0.45, 0.67)], with dimensions of [width and height (10, 20)], [target description: wearing a red shirt waving, with a dog next to him]".

[0097] S406. Sort the fixed prompt word templates for all targets in descending order of confidence level.

[0098] Among them, the confidence level represents the degree to which the detected region is a real target.

[0099] S408. Input the target image size information, global description text, and sorted fixed prompt word template into the text image model, and output the target image.

[0100] Among them, the Text-to-Image Model is a type of artificial intelligence model whose core capability is to automatically generate corresponding image content based on the input text description.

[0101] In the aforementioned image processing method, the size information and global description text of the target image sent by edge nodes, as well as the category information, confidence level, coordinate information, and attribute description text of any target in the target image, are received. For any target, the target's category information, confidence level, coordinate information, and attribute description text are embedded into a fixed prompt word template. The fixed prompt word templates of all targets are sorted in descending order of confidence level. The size information and global description text of the target image, along with the sorted fixed prompt word templates, are input into a text-based image model, and the target image is output. The method provided in this embodiment, by embedding the data sent by the received edge nodes into fixed prompt word templates and sorting the fixed prompt word templates according to confidence level before inputting them into a text-based image model, can maximize the restoration of the target image and reduce restoration errors.

[0102] In one exemplary embodiment, a third image processing method is provided, which includes the following:

[0103] (1) The edge side receives the input image, obtains the image size information, and uses small models such as YOLO to perform target detection on the image to obtain target information, including target category, confidence level, target position, and coordinates of the upper left corner of the detection box.

[0104] (2) At the same time, the input image is transmitted to multimodal models such as ViLBERT to obtain the semantic description information of the image, including the overall description of the image and the attribute description of each target.

[0105] (3) Package the target detection information, image size information and semantic description information into structured data (JSON format) and send it to the cloud.

[0106] (4) Receive data in the cloud, use a fixed template to embed target detection information and semantic description information into Prompt text prompts, convert them into text information that the large model can understand, and sort the target descriptions according to the confidence scores.

[0107] (5) The large model is based on text prompts. It uses a pre-trained text-to-image model to restore semantic information into images and then performs subsequent task processing, including behavior judgment, search, decision-making, etc.

[0108] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0109] Based on the same inventive concept, this application also provides an image processing apparatus for implementing the image processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more image processing apparatus embodiments provided below can be found in the limitations of the image processing method described above, and will not be repeated here.

[0110] In one exemplary embodiment, such as Figure 5 As shown, an image processing apparatus 500 is provided, including: a first receiving module 501, a first input module 502, a second input module 503, and a transmitting module 504, wherein:

[0111] The first receiving module 501 is used to receive the target image sent by the image acquisition device and obtain the size information of the target image.

[0112] The first input module 502 is used to input the target image into the target detection model and output the category information, confidence level and coordinate information of any target in the target image.

[0113] The second input module 503 is used to input the target image into the multimodal model and output the global description text of the target image, the first text length of the global description text, the attribute description text of any target, and the second text length of the attribute description text.

[0114] The sending module 504 is used to send the size information, the category information, the confidence level, the coordinate information, the global description text, and the attribute description text to the cloud based on the first text length, the second text length, and the category information.

[0115] In some embodiments, the sending module 504 is further configured to convert the size information, the category information, the confidence level, the coordinate information, the global description text, and the attribute description text into structured data; obtain a first number of bits of the structured data based on the first text length, the second text length, and the category information according to a first preset encoding rule; and send the structured data to the cloud based on the first number of bits.

[0116] In some embodiments, the sending module 504 is further configured to: obtain a first data size of the structured data based on the first number of bits; and obtain a first transmission delay required to transmit the structured data to the cloud based on the first data size; obtain a second transmission delay required to transmit the image data of the target image to the cloud; send the attribute data to the cloud if the first transmission delay is greater than the second transmission delay; and send the structured data to the cloud if the first transmission delay is not greater than the second transmission delay.

[0117] In some embodiments, the sending module 504 is further configured to obtain the size information, compression ratio, number of color channels, and number of channel bits of any color channel of the target image; according to a second preset encoding rule, obtain the second number of bits of the image data based on the size information, the compression ratio, the number of channels, and the number of channel bits, and obtain the second transmission delay based on the second number of bits.

[0118] In some embodiments, the image processing apparatus 500 is specifically used in the multimodal model to include a fixed question template that restricts the output rules for descriptive text.

[0119] In one exemplary embodiment, such as Figure 6 As shown, an image processing apparatus 600 is provided, including: a second receiving module 601, an embedding module 602, a sorting module 603, and a third input module 604, wherein:

[0120] The second receiving module 601 is used to receive the size information and global description text of the target image sent by the edge node, as well as the category information, confidence level, coordinate information and attribute description text of any target in the target image.

[0121] The embedding module 602 is used to embed the category information, confidence level, coordinate information and attribute description text of any target into a fixed prompt word template.

[0122] The sorting module 603 is used to sort the fixed prompt word templates of all targets in descending order of confidence level.

[0123] The third input module 604 is used to input the size information of the target image, the global description text, and the sorted fixed prompt word template into the text-to-image model, and output the target image.

[0124] Each module in the aforementioned image processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0125] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 7 As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores image data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements an image processing method.

[0126] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0127] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0128] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0129] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0130] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0131] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0132] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0133] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An image processing method, characterized in that, Applied to edge nodes, the method includes: Receive the target image sent by the image acquisition device, and obtain the size information of the target image; The target image is input into the target detection model, which outputs the category information, confidence score, and coordinate information of any target in the target image. The target image is input into a multimodal model, which outputs the global description text of the target image, the first text length of the global description text, the attribute description text of any target, and the second text length of the attribute description text. Based on the first text length, the second text length, and the category information, the size information, the category information, the confidence level, the coordinate information, the global description text, and the attribute description text are sent to the cloud.

2. The method according to claim 1, characterized in that, The step of sending the size information, category information, confidence score, coordinate information, global description text, and attribute description text to the cloud based on the first text length, the second text length, and the category information includes: The size information, category information, confidence level, coordinate information, global description text, and attribute description text are converted into structured data. According to the first preset encoding rule, the first number of bits of the structured data is obtained based on the first text length, the second text length, and the category information; Based on the first number of bits, the structured data is sent to the cloud.

3. The method according to claim 2, characterized in that, The step of sending the structured data to the cloud based on the first number of bits includes: Based on the first number of bits, obtain the first data size of the structured data, and based on the first data size, obtain the first transmission delay required to transmit the structured data to the cloud; Obtain the second transmission delay required to transmit the image data of the target image to the cloud; If the first transmission delay is greater than the second transmission delay, the image data will be sent to the cloud. If the first transmission delay is not greater than the second transmission delay, the structured data is sent to the cloud.

4. The method according to claim 3, characterized in that, The step of obtaining the second transmission delay required to transmit the image data of the target image to the cloud includes: Obtain the target image's size information, compression ratio, number of color channels, and number of bits per color channel; According to the second preset encoding rule, the second number of bits of the image data is obtained based on the size information, the compression ratio, the number of channels, and the number of channel bits, and the second transmission delay is obtained based on the second number of bits.

5. The method according to claim 1, characterized in that, The multimodal model includes a fixed question template that restricts the output rules of the descriptive text.

6. An image processing method, characterized in that, Applied to the cloud, the method includes: Receive the size information and global description text of the target image sent by the edge node, as well as the category information, confidence level, coordinate information and attribute description text of any target in the target image; For any given target, the target's category information, confidence level, coordinate information, and attribute description text are embedded into a fixed prompt word template; Sort the fixed prompt word templates for all targets in descending order of confidence level; The target image's size information, global description text, and sorted fixed prompt word template are input into the text-based image model, and the target image is output.

7. An image processing apparatus, characterized in that, The device includes: The first receiving module is used to receive the target image sent by the image acquisition device and obtain the size information of the target image; The first input module is used to input the target image into the target detection model and output the category information, confidence score and coordinate information of any target in the target image; The second input module is used to input the target image into the multimodal model and output the global description text of the target image, the first text length of the global description text, the attribute description text of any target, and the second text length of the attribute description text. The sending module is used to send the size information, the category information, the confidence level, the coordinate information, the global description text, and the attribute description text to the cloud based on the first text length, the second text length, and the category information.

8. An image processing apparatus, characterized in that, The device includes: The second receiving module is used to receive the size information and global description text of the target image sent by the edge node, as well as the category information, confidence level, coordinate information and attribute description text of any target in the target image; An embedding module is used to embed the category information, confidence level, coordinate information, and attribute description text of any target into a fixed prompt word template. The sorting module is used to sort the fixed prompt word templates of all targets in descending order of confidence level; The third input module is used to input the size information of the target image, the global description text, and the sorted fixed prompt word template into the text-to-image model, and output the target image.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images