Visual task processing method and device, equipment and storage medium

Abstract

The invention relates to a visual task processing method and device, equipment and a storage medium, and relates to the technical field of artificial intelligence. The method comprises the following steps: dividing a target image to be coded into at least two image blocks; determining a key block from the at least two image blocks according to the pooling information of each image block; and encoding the key blocks to obtain a feature code stream, and sending the feature code stream to an edge end. By adopting the method, the visual task processing efficiency can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a visual task processing method, device, equipment and storage medium, which can be applied to the field of machine vision technology. Background Art

[0002] The visual Internet of Things is an upgraded version of the traditional Internet of Things. It uses various visual sensors, including cameras, mobile phones, surveillance cameras, etc., to collect images of objects, people, vehicles, etc., and uses image processing algorithms and machine vision tasks to calculate and process these data to obtain valuable information from the images. The information is then transmitted to the application end (such as the edge end) through the network, and after post-processing and intelligent algorithm analysis at the application end, communication and information exchange are achieved.

[0003] With the rapid development of 5G, big data, and artificial intelligence, visual tasks in the field of visual IoT are becoming increasingly popular. Common visual tasks include image classification, video object detection, and image segmentation. The processing of visual tasks in the field of visual IoT usually involves image encoding and decoding technology. However, due to the limitation of the communication bandwidth of the visual IoT, the existing image encoding and decoding technology cannot meet the needs of fast response to visual tasks. Summary of the invention

[0004] The embodiments of the present application provide a visual task processing method, apparatus, computer equipment and storage medium, which can improve the efficiency of visual task processing.

[0005] In a first aspect, an embodiment of the present application provides a visual task processing method, which is applied to a terminal, and the method includes:

[0006] Dividing a target image to be encoded into at least two image blocks;

[0007] Determining a key block from the at least two image blocks according to the pooling information of each image block;

[0008] The key blocks are encoded to obtain a feature code stream, and the feature code stream is sent to the edge end.

[0009] In one of the embodiments, determining the key block from the at least two image blocks according to the pooling information of each image block includes:

[0010] Determine a combined pooling value of each image block according to the pixel value of each pixel point contained in each image block;

[0011] A key block is determined from the at least two image blocks according to the combined pooled value of each image block.

[0012] In one of the embodiments, determining the combined pooling value of each image block according to the pixel value of each pixel point included in each image block includes:

[0013] For each image block, determine the maximum pooling value and the average pooling value of the image block according to the pixel value of each pixel point contained in the image block;

[0014] Using the preset pooling ratio, the maximum pooling value and the average pooling value of the image block are weighted summed to obtain the combined pooling value of the image block.

[0015] In one of the embodiments, determining the key block from the at least two image blocks according to the combined pooling value of each image block includes:

[0016] Determine the number of non-critical blocks according to a preset compression ratio and the number of image blocks;

[0017] A key block is determined from the at least two image blocks according to the number of non-key blocks and the combined pooling value of each image block.

[0018] In one of the embodiments, determining the key block from the at least two image blocks according to the number of non-key blocks and the combined pooling value of each image block comprises:

[0019] sorting the at least two image blocks according to the combined pooling value of each image block;

[0020] Selecting a non-key block from the at least two image blocks according to the number of the non-key blocks and the sorting result;

[0021] The image blocks other than the non-key blocks in the at least two image blocks are used as key blocks.

[0022] In one embodiment, encoding the key blocks to obtain a feature code stream includes:

[0023] Encoding the key blocks to obtain encoded visual blocks;

[0024] Information is embedded according to the positions of the coded visual blocks and the non-critical blocks in the target image to generate a feature code stream.

[0025] In one embodiment, encoding the key blocks to obtain the encoded visual blocks includes:

[0026] Performing linear projection on the key block to obtain block embedding information of the key block;

[0027] The block embedding information of the key block and the position embedding information of the key block in the target image are encoded by a visual encoder to obtain an encoded visual block corresponding to the key block.

[0028] In one embodiment, the method further comprises:

[0029] Receiving a trained visual encoder sent by the cloud; wherein the trained visual encoder is obtained by training the cloud based on a training data set updated by the received decoded visual block; and the decoded visual block is obtained by decoding the edge end based on the feature code stream;

[0030] Use the trained visual encoder to replace the local visual encoder.

[0031] In one embodiment, the method further comprises:

[0032] Receive the target visual task processing request sent by the edge end;

[0033] Based on the target visual task processing request, a target image to be encoded is obtained.

[0034] In one of the embodiments, the target visual task processing request is generated by the edge terminal based on an original visual task processing request sent by the cloud terminal; the original visual task processing request is generated by the cloud terminal based on a task call request sent by the user terminal.

[0035] In a second aspect, an embodiment of the present application provides a visual task processing method, which is applied to an edge terminal, and the method includes:

[0036] A feature code stream sent by a receiving terminal; wherein the feature code stream is obtained by encoding a key block by the terminal, and the key block is determined from each image block by the terminal according to pooling information of each image block obtained by dividing a target image to be encoded;

[0037] The feature code stream is decoded to obtain a decoded visual block.

[0038] In one of the embodiments, the feature code stream is generated by the terminal according to the encoded visual block obtained by encoding the key block, and the position embedding information of the non-key block in the target image;

[0039] Correspondingly, decoding the feature code stream to obtain a decoded visual block includes:

[0040] Acquire the position embedding information of the coded visual block and the non-critical block in the target image from the feature code stream;

[0041] The position embedding information of the coded visual block and the non-critical sub-block in the target image is decoded by a visual decoder to obtain a decoded visual block.

[0042] In one embodiment, the method further comprises:

[0043] Sending the decoded visual block to the cloud; wherein the decoded visual block is used for the cloud to update the training data set;

[0044] Receiving a trained visual decoder sent by the cloud; wherein the trained visual decoder is trained based on an updated training data set;

[0045] Use the trained visual decoder to replace the local visual decoder.

[0046] In one embodiment, the method further comprises:

[0047] Receiving an original visual task processing request sent by a cloud; wherein the original visual task processing request is generated by the cloud based on a task call request sent by a user terminal;

[0048] Based on the original visual task processing request, sending a target visual task processing request to the terminal; wherein the target visual task processing request is used to request the terminal to feedback the feature code stream;

[0049] Accordingly, the method further includes:

[0050] According to the decoded visual block, a visual task is performed to obtain a task processing result;

[0051] Feedback the task processing result to the user terminal.

[0052] In a third aspect, an embodiment of the present application provides a visual task processing method, which is applied to the cloud, and the method includes:

[0053] Receive the task call request sent by the user;

[0054] Based on the task calling request, sending an original visual task processing request to the edge end;

[0055] Among them, the original visual task processing request is used to instruct the edge end to send a target visual task processing request to the terminal; the target visual task processing request is used to instruct the terminal to determine the key blocks from each image block according to the pooling information of each image block obtained by dividing the target image to be encoded, and encode the key blocks to obtain a feature code stream, and the edge end determines the task processing result according to the feature code stream.

[0056] In one embodiment, sending the original visual task processing request to the edge end based on the task calling request includes:

[0057] Acquire user information associated with the user terminal according to the task call request;

[0058] Performing authority verification on the user information;

[0059] If the authority verification is passed, the original visual task processing request is sent to the edge.

[0060] In one embodiment, the method further comprises:

[0061] Receiving a decoded visual block sent by the edge end;

[0062] Using the decoded visual block, updating the training data set;

[0063] When the updated training data set meets the training conditions, the visual encoder and / or the visual decoder are trained using the updated training data set;

[0064] The trained visual decoder is sent to the edge end, and / or the trained visual encoder is sent to the terminal.

[0065] In a fourth aspect, an embodiment of the present application provides a visual task processing device, which is configured in a terminal, and the device includes:

[0066] An image division module, used for dividing a target image to be encoded into at least two image blocks;

[0067] A key block determination module, configured to determine a key block from the at least two image blocks according to the pooling information of each image block;

[0068] An encoding module, used for encoding the key blocks to obtain a feature code stream;

[0069] The code stream sending module is used to send the characteristic code stream to the edge end.

[0070] In a fifth aspect, an embodiment of the present application provides a visual task processing device, which is configured in an edge terminal, and the device includes:

[0071] A code stream receiving module, used for receiving a characteristic code stream sent by a terminal; wherein the characteristic code stream is obtained by encoding a key block by the terminal, and the key block is determined from each image block by the terminal according to pooling information of each image block obtained by dividing a target image to be encoded;

[0072] The decoding module is used to decode the feature code stream to obtain a decoded visual block.

[0073] In a sixth aspect, an embodiment of the present application provides a visual task processing device, which is configured in the cloud, and the device includes:

[0074] A request receiving module, used to receive a task calling request sent by a user terminal;

[0075] A request sending module, used for sending an original visual task processing request to the edge end based on the task call request;

[0076] Among them, the original visual task processing request is used to instruct the edge end to send a target visual task processing request to the terminal; the target visual task processing request is used to instruct the terminal to determine the key blocks from each image block according to the pooling information of each image block obtained by dividing the target image to be encoded, and encode the key blocks to obtain a feature code stream, and the edge end determines the task processing result according to the feature code stream.

[0077] In a seventh aspect, an embodiment of the present application provides a terminal device, including: a first processor and a first transmitter;

[0078] The first processor is used to divide the target image to be encoded into at least two image blocks, and determine a key block from the at least two image blocks according to pooling information of each image block, and encode the key block to obtain a feature code stream;

[0079] The first transmitter is used to send the feature code stream to the edge end.

[0080] In an eighth aspect, an embodiment of the present application provides an edge device, including: a first receiver and a second processor;

[0081] The first receiver is used to receive a feature code stream sent by a terminal; wherein the feature code stream is obtained by the terminal encoding a key block, and the key block is determined from each image block by the terminal according to pooling information of each image block obtained by dividing a target image to be encoded;

[0082] The second processor is used to decode the feature code stream to obtain a decoded visual block.

[0083] In a ninth aspect, an embodiment of the present application provides a cloud device, including: a second receiver and a second transmitter;

[0084] The second receiver is used to receive a task calling request sent by a user terminal;

[0085] The second transmitter is used to send an original visual task processing request to the edge end based on the task calling request;

[0086] Among them, the original visual task processing request is used to instruct the edge end to send a target visual task processing request to the terminal; the target visual task processing request is used to instruct the terminal to determine the key blocks from each image block according to the pooling information of each image block obtained by dividing the target image to be encoded, and encode the key blocks to obtain a feature code stream, and the edge end determines the task processing result according to the feature code stream.

[0087] In a tenth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the following steps are implemented:

[0088] Dividing a target image to be encoded into at least two image blocks;

[0089] Determining a key block from the at least two image blocks according to the pooling information of each image block;

[0090] The key blocks are encoded to obtain a feature code stream, and the feature code stream is sent to the edge end.

[0091] In an eleventh aspect, an embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0092] A feature code stream sent by a receiving terminal; wherein the feature code stream is obtained by encoding a key block by the terminal, and the key block is determined from each image block by the terminal according to pooling information of each image block obtained by dividing a target image to be encoded;

[0093] The feature code stream is decoded to obtain a decoded visual block.

[0094] In a twelfth aspect, an embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the following steps are implemented:

[0095] Receive the task call request sent by the user;

[0096] Based on the task calling request, sending an original visual task processing request to the edge end;

[0097] Among them, the original visual task processing request is used to instruct the edge end to send a target visual task processing request to the terminal; the target visual task processing request is used to instruct the terminal to determine the key blocks from each image block according to the pooling information of each image block obtained by dividing the target image to be encoded, and encode the key blocks to obtain a feature code stream, and the edge end determines the task processing result according to the feature code stream.

[0098] The above-mentioned visual task processing method, device, equipment and storage medium, by determining the key blocks from the multiple image blocks obtained by dividing the target image according to the pooling information of each image block obtained by dividing the target image, and only encode the determined key blocks to obtain the feature code stream, and then send the feature code stream to the edge end to execute the subsequent process of visual task processing. The above-mentioned scheme, by introducing pooling information for characterizing pixel changes, can accurately extract the key pixel area with rich image information in the target image, that is, the key block, and only encode the key block, thereby achieving the effect of selectively encoding the key pixel area in the image data. Compared with encoding all pixel points in the target image, the amount of data of the encoded feature code stream is reduced, thereby improving the efficiency of transmitting the feature code stream to the edge end, thereby improving the efficiency of the entire visual task processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0099] Figure 1 is an application environment diagram of a visual task processing method in one embodiment;

[0100] Figure 2 is a flowchart of a visual task processing method in one embodiment;

[0101] Figure 3 A schematic diagram of a process of image segmentation in one embodiment;

[0102] Figure 4 A schematic diagram of a process for determining key blocks in one embodiment;

[0103] Figure 5 A schematic diagram of a flow chart for determining key blocks in another embodiment;

[0104] Figure 6 A schematic diagram of a process of generating a feature code stream in one embodiment;

[0105] Figure 7 is a flowchart of visual encoding in one embodiment;

[0106] Figure 8 is a flow chart of a visual task processing method in another embodiment;

[0107] Fig. 9 A schematic diagram of a process of decoding a feature code stream in one embodiment;

[0108] Fig.10 is a flowchart of visual decoding in one embodiment;

[0109] Fig.11 is a flow chart of a visual task processing method in another embodiment;

[0110] Fig.12 A schematic diagram of a process of training a codec in one embodiment;

[0111] Fig.13 is an architecture diagram of a visual task processing system in one embodiment;

[0112] Fig.14 is a signaling diagram of a visual task processing method in one embodiment;

[0113] Fig.15 is a structural block diagram of a visual task processing device in one embodiment;

[0114] Fig.16 is a structural block diagram of a visual task processing device in another embodiment;

[0115] Fig.17 is a structural block diagram of a visual task processing device in another embodiment;

[0116] Fig.18 An internal structure diagram of an edge terminal or terminal device in one embodiment;

[0117] Fig.19 1 is a diagram of the internal structure of a cloud device in one embodiment. DETAILED DESCRIPTION

[0118] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0119] Figure 1 A schematic diagram of an application scenario of a visual task processing method provided in an embodiment of the present application. The visual task may be, but is not limited to, target detection / recognition (e.g., face detection, vehicle recognition, etc.), defect detection, etc. The processing of visual tasks generally involves three stages: encoding, decoding, and detection / recognition. Figure 1 As shown, the application scenario includes a terminal 100 with an encoding function, and an edge terminal 200 with a decoding function and a detection / recognition function. The terminal 100 can interact with the edge terminal 200 through a network. For example, the terminal 100 can send a code stream obtained by encoding an image to the edge terminal 200 through the network.

[0120] In one embodiment, the application scenario may also include a cloud 300. The cloud 300 may be used to dock with the user terminal 400 to obtain the user's visual-oriented task call request; the cloud 300 may also be used to dock with the edge 200, and based on the user's task call request, trigger the edge 200 to start the processing flow of the visual task. For example, the cloud 300 triggers the edge 200 to interact with the terminal 100, so that the terminal 200 executes the process of encoding the image, and then the edge 200 decodes, detects / identifies, and performs a series of operations on the code stream obtained by the terminal 100 through the encoding technology, thereby completing the processing of the entire visual task. It is understandable that the cloud 300 can also dock with the terminal 100 to directly trigger the terminal 100 to perform the encoding process of the relevant visual task, but considering that the cloud 300 directly communicates with the terminal 100, the communication cost required is relatively large, so the embodiment of the present application uses the edge 200 to trigger the terminal 100 to perform the encoding process of the relevant visual task.

[0121] It should be noted that in the process of processing visual tasks, when traditional technology encodes an image, it encodes all the pixels in the image, resulting in a huge amount of generated code stream data, which in turn affects the efficiency of visual task processing.

[0122] Based on the above-mentioned traditional technology, an embodiment of the present application provides a visual task processing method, which improves the efficiency of visual task processing by selectively encoding pixel points in an image.

[0123] It should be further explained that the beneficial effects brought about by the embodiments of the present application or the technical problems solved are not limited to this one, but may also include other implicit or related problems. For details, please refer to the description of the following embodiments.

[0124] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0125] In one embodiment, Figure 2 As shown, a visual task processing method is provided, which is applied to Figure 1 The method mainly involves the encoding stage in visual task processing, and specifically includes the following steps:

[0126] S201, dividing a target image to be encoded into at least two image blocks.

[0127] The so-called target image is the image that needs to be encoded. Optionally, in a visual task processing scenario, the target image can be an image corresponding to the visual task to be processed; for example, if the visual task is a face recognition task, the target image can be a face image; if the visual task is a defect detection task, the target image can be an image of an object to be detected.

[0128] Optionally, there are many ways for the terminal to obtain the target image to be encoded, which is not limited in this embodiment. For example, an image acquisition module may be integrated in the terminal, and the target image to be encoded may be obtained through the image acquisition module; or the terminal may also obtain the target image to be encoded from a local image library; or the terminal may also control an image acquisition device in the visual Internet of Things to acquire the target image, etc.

[0129] Optionally, the target image to be encoded may be divided into two or more image blocks by non-overlapping uniform division. For example, if the size of the target image is n*n, the target image may be divided into m*m image blocks of the same size by non-overlapping uniform division. The size of each image block is Take the target image size of 28*28 as an example. Figure 3 As shown, it is evenly divided into 4*4 image blocks in a non-overlapping manner, and the size of each image block is 7*7. After the target image is divided into 16 image blocks of the same size, an identification information is assigned to each image block in turn according to the position of the image block in the target image, for example, the identification information of image block 1 is 1, etc.

[0130] S202: Determine a key block from at least two image blocks according to the pooling information of each image block.

[0131] Among them, the so-called pooling information can be used to characterize the pixel changes of an image block. Optionally, the pixel changes of an image block can reflect the richness of the image information contained in the image block. In one implementation, the pooling information can be presented in the form of a pooling value, that is, a numerical value. At this time, the larger the pooling value, the greater the pixel change of the image block; the greater the pixel change of the image block, the richer the image information contained in the image block.

[0132] The so-called key blocks are image blocks with relatively rich image information. Optionally, the number of key blocks can be one or more.

[0133] Optionally, after obtaining at least two image blocks, a pre-set pooling method may be used to perform pooling processing on each image block to obtain pooling information of each image block; and then based on the pooling information of each image block, a key block may be selected from the at least two image blocks obtained. For example, when the pooling information is presented as a pooling value, the pooling information of each image block may be compared with a set pooling threshold, and the image block whose pooling information is greater than the set pooling threshold is used as the key block.

[0134] S203, encode the key blocks to obtain a feature code stream, and send the feature code stream to the edge end.

[0135] In the embodiment of the present application, the characteristic code stream may be a bit code stream, or a code stream in other forms, etc. Optionally, the form of the characteristic code stream may be determined by the encoding method and the code stream transmission method supported by the communication channel between the terminal and the edge end.

[0136] Optionally, a pre-built visual encoder may be used to encode the key blocks. In one implementation, different visual encoders correspond to different visual tasks. In this case, the required visual encoder may be selected according to the visual task, and then the selected visual encoder may be used to encode the key blocks.

[0137] Furthermore, the encoded feature code stream is sent to the edge through the network, so that the edge can perform the decoding and detection / recognition phases in the visual task processing based on the feature code stream, thereby completing the processing of the entire visual task.

[0138] In the above-mentioned visual task processing method, the key blocks are determined from the multiple image blocks obtained by dividing the target image according to the pooling information of each image block obtained by dividing the target image, and only the determined key blocks are encoded to obtain the feature code stream, and then the feature code stream is sent to the edge end to execute the subsequent process of visual task processing. The above-mentioned scheme, by introducing pooling information for characterizing pixel changes, can accurately extract the key pixel area with rich image information in the target image, that is, the key block, and only encode the key block, thereby achieving the effect of selectively encoding the key pixel area in the image data. Compared with encoding all pixel points in the target image, the amount of data of the encoded feature code stream is reduced, thereby improving the efficiency of transmitting the feature code stream to the edge end, thereby improving the efficiency of the entire visual task processing.

[0139] Optionally, compared to single maximum pooling and mean pooling, mixed pooling is a pooling strategy that takes advantage of both maximum pooling and mean pooling, and can more accurately reflect the image information richness of image blocks to a certain extent. Therefore, this embodiment provides an optional method for determining key blocks based on combined pooling values. Figure 4 As shown, the implementation process may include the following steps:

[0140] S401, determining a combined pooling value of each image block according to the pixel value of each pixel point included in each image block.

[0141] Among them, the combined pooling value is a presentation form of pooling information.

[0142] Optionally, based on the logic of the combined pooling method, the combined pooling value of each image block may be determined according to the pixel value of each pixel point included in each image block.

[0143] For example, for each image block, the maximum pooling value and the average pooling value of the image block are determined according to the pixel values ​​of each pixel point contained in the image block; the preset pooling ratio is used to perform weighted summation on the maximum pooling value and the average pooling value of the image block to obtain the combined pooling value of the image block. Among them, the preset pooling ratio can be flexibly adjusted according to the actual visual task; optionally, the preset pooling ratio can include a preset average pooling ratio and a preset maximum pooling ratio; further, the sum of the preset average pooling ratio and the preset maximum pooling ratio is equal to 1.

[0144] For example, for each image block, the average value of the pixel values ​​of each pixel point contained in the image block can be used as the average pooling value. For example, when the size of the target image is n*n, the target image is divided into m*m image blocks of the same size, and the size of each image block is In the case of , for each image block, the average pooling value p of the image block can be determined by combining the following formula (1).

[0145]

[0146] Wherein, f(x, y) represents the pixel value of each pixel in the image block.

[0147] Optionally, the maximum value of the pixel values ​​of each pixel point included in the image block can be used as the maximum pooling value. For example, the maximum pooling value g of the image block can be determined in combination with the following formula (2).

[0148]

[0149] Furthermore, the average pooling value p is weighted by the preset average pooling ratio u; the maximum pooling value g is weighted by the preset maximum pooling ratio 1-u; and then the weighted sum of the two is used as the combined pooling value of the image block. That is, the combined pooling value of the image block can be expressed as:

[0150] l(i, j) = u*p + (1-u)*p (3)

[0151] Wherein, l(i, j) is the combined pooling value calculated based on combined pooling for all pixels in the image block. The value range of i and j is 0 to m-1.

[0152] Based on this, the above method of determining the combined pooling value is performed on each image block, that is, the combined pooling value of all image blocks can be obtained. Figure 3 Each image block shown uses the above method to determine the combined pooling value, that is, Figure 3 The combined pooling values ​​of all image patches in .

[0153] S402: Determine a key block from at least two image blocks according to the combined pooling value of each image block.

[0154] Optionally, after determining the combined pooling value of each image block, the combined pooling value of each image block may be compared with a preset pooling threshold, and the image block having a combined pooling value greater than the set pooling threshold may be taken as a key block.

[0155] Alternatively, a preset number of image blocks may be selected from at least two image blocks as key blocks according to the combined pooling value of each image block. For example, at least two image blocks may be sorted in descending order according to the combined pooling value of each image block; then, a preset number of image blocks in the front order are selected as key blocks.

[0156] It can be understood that, by introducing the combined pooling value in this embodiment, key blocks with rich image information in the target image can be accurately extracted, providing data support for selectively encoding key pixel areas in the image.

[0157] Optionally, the compression ratio used in encoding is an important feature of the encoding process. At present, the compression ratio of the encoder designed based on the deep convolutional neural network structure cannot be adjusted, resulting in inflexible encoding. Based on this, on the basis of the above embodiment, the embodiment of the present application provides an optional method for determining key blocks based on the compression ratio. Figure 5 As shown, the implementation process may include the following steps:

[0158] S501, determining the number of non-critical blocks according to a preset compression ratio and the number of image blocks.

[0159] Among them, corresponding to the key blocks, the so-called non-key blocks are image blocks with little pixel change, or image information is not rich.

[0160] In the embodiment of the present application, the preset compression ratio can be flexibly configured according to the actual visual task. Optionally, the relationship between the preset compression ratio, the number of image blocks, and the number of non-critical blocks can be expressed by the following formula (4).

[0161]

[0162] Among them, α represents the preset compression ratio, m 2 (ie, m*m) represents the number of image blocks, and q represents the number of non-critical blocks.

[0163] Furthermore, after determining the preset compression ratio and the number of image blocks, the preset compression ratio and the number of image blocks can be substituted into formula (4) for solution to obtain the number of non-critical blocks.

[0164] S502: Determine a key block from at least two image blocks according to the number of non-key blocks and the combined pooling value of each image block.

[0165] One possible implementation method is to use the difference between the number of image blocks and the number of non-critical blocks as the number of critical blocks; and select image blocks with the determined number of critical blocks from at least two image blocks as critical blocks based on the combined pooling value of each image block.

[0166] Another possible implementation is to sort at least two image blocks according to the combined pooling value of each image block; select non-critical blocks from at least two image blocks according to the number of non-critical blocks and the sorting result; and use image blocks other than the non-critical blocks in at least two image blocks as critical blocks. For example, at least two image blocks can be sorted in descending order according to the combined pooling value of each image block; according to the sorting result, a preset number of image blocks in the latter order are used as non-critical blocks, where the preset number is the number of non-critical blocks; and image blocks other than the non-critical blocks in all image blocks are used as critical blocks.

[0167] It can be understood that the compression ratio is introduced in this embodiment to control the number of non-critical blocks in the image, and then control the number of critical blocks in the image, thereby increasing the flexibility of the encoding process; further, the compression ratio in this embodiment can be flexibly adjusted according to actual visual tasks, thereby broadening the encoding scenarios.

[0168] Based on the above embodiments, in one embodiment, Figure 6 As shown, the embodiment of the present application provides an optional method for encoding key blocks. Figure 6 , the implementation process may include the following steps:

[0169] S601, encoding the key blocks to obtain encoded visual blocks.

[0170] In one implementation, for each key block, a patch embedding process (i.e., Patch Embeddings) may be performed on the key block to obtain the patch embedding information of the key block; then, the patch embedding information of the key block may be input into a pre-built visual encoder, which is then encoded to obtain the encoded visual block of the key block. The patch embedding information is used to express relevant information of the image in the key block.

[0171] In order to obtain a coded visual block with richer image information, another feasible method is to perform linear projection on the key block to obtain the block embedding information of the key block; through the visual encoder, the block embedding information of the key block and the position embedding information of the key block in the target image are encoded to obtain the coded visual block corresponding to the key block.

[0172] Among them, the visual encoder in this embodiment is a Transformer-based encoder. The so-called linear projection is the process of converting the key block from a two-dimensional image space to a one-dimensional space; optionally, the encoding of the block embedding information is essentially the encoding of the image information contained in the key block. In the application embodiment, the result of encoding the block embedding information is called block encoding data. The so-called position embedding information of the key block in the target image is used to characterize the position of the key block in the target image, for example, it can be the identification information of the key block.

[0173] Optionally, for each key block, a linear projection can be performed on the key block to obtain the block embedding information of the key block; then, the block information of the key block is spliced ​​with the position embedding information of the key block in the target image to obtain the encoding information of the key block. Then, the encoding information of all key blocks is input into the visual encoder together to obtain the encoded visual block corresponding to each key block. Obviously, the encoded visual block corresponding to each key block includes the encoding results of two dimensions: position and block embedding information.

[0174] For example, Figure 7As shown, for the 16 image blocks corresponding to the target image, after selecting key blocks from the 16 image blocks based on the combined pooling value of each image block, namely image block 2, image block 7, image block 9, image block 14 and image block 16, the encoding information of each key block can be input into the visual encoder (ie, VisionTransformer) to obtain the encoded visual blocks (Encoded Visible Patches) corresponding to each key block. Figure 7 The white boxes in the figure represent key blocks, the boxes filled with slashes are the encoded visual blocks corresponding to the key blocks (i.e., Encoded Visible Patches), and the black boxes represent non-key blocks in the target image. Since the non-key blocks are not encoded, the non-key blocks here are mask blocks (i.e., Mask Tokens), and their block encoding data can be random values ​​or preset fixed values.

[0175] It can be understood that in this embodiment, by introducing block embedding information and position embedding information for encoding, the visual encoder can extract richer information in the key blocks.

[0176] S602, embedding information according to the positions of the coded visual blocks and the non-critical blocks in the target image to generate a feature code stream.

[0177] Optionally, the coded visual blocks and the position embedding information of the non-critical blocks in the target image may be concatenated to obtain a feature code stream.

[0178] Or, for non-critical chunks, Figure 7 For the image blocks other than image block 2, image block 7, image block 9, image block 14 and image block 16, a fixed value can be assigned to each non-critical block; then, the encoded visual block, as well as the position embedding information of the non-critical block in the target image and the assigned fixed value can be spliced ​​to obtain the feature code stream.

[0179] It can be understood that in this embodiment, the position of the non-critical block in the target image is introduced into the feature code stream to embed information, so as to facilitate the edge end to quickly reconstruct the target image.

[0180] In one embodiment, in order to ensure the accuracy of terminal encoding, the visual encoder in the terminal needs to be updated regularly. Optionally, the terminal receives a trained visual encoder sent by the cloud; wherein the trained visual encoder is trained by the cloud based on a training data set updated by the received decoded visual block; the trained visual encoder is used to replace the local visual encoder. wherein the decoded visual block is obtained by decoding the edge end based on the feature code stream.

[0181] Specifically, each time the edge receives a decoded visual block, it will send it to the cloud. The cloud will update the training data set based on the received decoded visual block. When the updated training data set meets the training conditions, the cloud will use the updated training data set to train the visual encoder and feed the trained visual encoder back to the terminal. After receiving the trained visual encoder sent by the cloud, the terminal will replace the local visual encoder with it, thus completing the update of the local visual encoder of the terminal. This achieves the effect of continuously updating the visual encoder deployed in the Internet of Things.

[0182] Optionally, in the scenario where the visual task processing method of the embodiment of the present application is applied to the cloud, terminal and edge in the visual Internet of Things to realize a certain type of specific visual task through three-party interaction, the terminal will receive the target visual task processing request sent by the edge; based on the target visual task processing request, the target image to be encoded is obtained. Among them, the target visual task processing request is generated by the edge based on the original visual task processing request sent by the cloud; the original visual task processing request is generated by the cloud based on the task call request sent by the user.

[0183] Specifically, in this embodiment, when the user has a task call demand, a task call request will be initiated to the cloud through the user end. The cloud will generate an original visual task processing request based on the task call request and send it to the edge end. The edge end can directly send the original visual task processing request as a target visual task processing request to the terminal, or it can also generate a new target visual task processing request after processing the original visual task processing request and send it to the terminal. For example, the original visual task processing request can be parsed to analyze the processing tasks that need to be performed by the terminal (such as image acquisition and visual encoding tasks), and then the target visual task processing request can be regenerated based on the processing task and sent to the terminal. After the terminal receives the target visual task processing request, it will obtain the target image to be encoded corresponding to the target visual task processing request.

[0184] In one embodiment, Figure 8 As shown, a visual task processing method is provided, which is applied to Figure 1 The edge end 200 in the example is used as an example to illustrate, and the edge end 200 is mainly used to perform the decoding part in the visual task. The following steps are included:

[0185] S801, receiving a feature code stream sent by a terminal.

[0186] The so-called feature code stream is obtained by encoding the key blocks by the terminal, and the key blocks are pooling information of each image block obtained by the terminal according to dividing the target image to be encoded, and are determined from each image block. It should be noted that the specific generation process of the feature code stream has been introduced in the above embodiment and will not be repeated here.

[0187] Optionally, since the visual processing task of the embodiment of the present application is a visual processing task based on the visual Internet of Things, the terminal in the above embodiment encodes the target image to be encoded, and after obtaining the feature code stream, the feature code stream will be transmitted to the edge end in the visual Internet of Things for performing the decoding task. Correspondingly, the edge end will receive the feature code stream sent by the terminal.

[0188] S802, decoding the feature code stream to obtain a decoded visual block.

[0189] The so-called decoded visual block is the decoding result of the feature code stream, which can be used to reconstruct the target image to be encoded.

[0190] Optionally, this embodiment can be a coded visual block obtained by parsing the terminal encoding the key blocks of the target image from the feature code stream, and then inputting the parsed coded visual block into a visual decoder that matches the visual encoder on the terminal side, and the visual decoder decodes the input coded visual block to obtain a decoded visual block. Specifically, the above embodiment has introduced that the terminal encodes the block embedding information of the key block and the position embedding information of the key block in the target image to generate the coded visual block; so when the visual decoder of this embodiment performs a decoding operation on the coded visual block, on the one hand, it parses the decoded visual block corresponding to each coded visual block, and on the other hand, based on the position embedding information corresponding to each coded visual block, parses the position of the non-key block in the target image, and decodes the decoded visual block corresponding to the non-key block according to the position of each non-key block and the block coding data of the coded visual block near the key block.

[0191] In the visual task processing method of the embodiment of the present application, the feature code stream received by the edge end is obtained by encoding the key blocks in the target image to be encoded by the terminal. Since the feature code stream only contains the visual coding data of the key blocks in the target image, this embodiment does not need to transmit the visual coding data of all pixels in the target image, which reduces the transmission amount and receiving efficiency of the feature code stream. In addition, when feature decoding is performed based on the feature code stream, the decoded visual block of the entire target image can be reconstructed based only on the visual coding data of the key blocks, which realizes the completion of the image decoding part in the visual task on the basis of transmitting a small amount of feature code stream, and provides a new solution for the interaction between the terminal and the edge end based on the visual Internet of Things to realize the image encoding and decoding operation in the visual task.

[0192] Optionally, when the visual task processing method of the present application is applied to a scenario in which the cloud, terminal and edge in the visual Internet of Things interact to complete a certain type of specific visual task (such as target recognition or target detection task), the edge will receive the original visual task processing request sent by the cloud, and based on the original visual task processing request, send a target visual task processing request to the terminal; wherein the original visual task processing request is generated by the cloud based on the task call request sent by the user; the target visual task processing request is used to request the terminal to feedback the feature code stream. Correspondingly, after the edge decodes the feature decoding stream to obtain the decoded visual block, it will also perform the visual task according to the decoded visual block, obtain the task processing result, and feedback the task processing result to the user.

[0193] Specifically, in this embodiment, when the user has a task call demand, a task call request will be initiated to the cloud through the user end. The cloud will generate an original visual task processing request based on the task call request and send it to the edge end. The edge end can directly send the original visual task processing request as a target visual task processing request to the terminal, or it can also process the original visual task processing request and generate a new target visual task processing request and send it to the terminal. For example, the original visual task processing request can be parsed to analyze the processing tasks that need to be performed by the terminal (such as image acquisition and visual encoding tasks), and then the target visual task processing request can be regenerated based on the processing task and sent to the terminal. After receiving the target visual task processing request, the terminal will obtain the target image to be encoded corresponding to the target visual task processing request, divide the target image into at least two image blocks, and determine the key blocks from the at least two image blocks according to the pooling information of each image block; encode the key blocks to obtain the feature code stream, and send the feature code stream as the response result of the target visual task to the edge end. Since the edge end has only completed the decoding part of the visual task after decoding the feature decoding stream to obtain the decoded visual block, it is also necessary to directly send the obtained decoded visual block to the downstream machine vision task for the visual Internet of Things, that is, continue to perform the downstream visual task based on the decoded visual block to obtain the task processing result. For example, it can be to call the visual algorithm corresponding to the visual task (such as the target recognition algorithm or the target detection algorithm) to perform visual task processing (such as target recognition processing or target detection processing) based on the target image reconstructed by the decoded visual block to obtain the task processing result of the visual task. Since the visual task is essentially generated based on the task call request of the user end, the edge end can directly or indirectly feedback the task processing result to the user end through the cloud after obtaining the task processing result. In order to provide visual task processing services for the user end. Optionally, the edge end can not only feed back the task processing results to the user end, but also perform statistical analysis of the visual task and display the analysis results, etc. There is no limitation on this.

[0194] In one embodiment, in order to ensure that the visual decoder can decode the decoded visual block corresponding to the target image more accurately, the terminal of the above embodiment can choose to generate the feature code stream based on the encoded visual block obtained by encoding the key block and the position embedding information of the non-key block in the target image. Fig. 9 and Fig.10 As shown, the edge end executes S802 to decode the feature code stream, and the specific steps of obtaining the decoded visual block may include:

[0195] S901, obtaining the position embedding information of the coded visual block and the non-critical block in the target image from the feature code stream.

[0196] Optionally, since the feature code stream of this embodiment is generated by the terminal based on the coded visual blocks obtained by encoding the key blocks and the position embedding information of the non-key blocks in the target image, the edge side parses the feature code stream to obtain the position embedding information of the coded visual blocks and the non-key blocks in the target image.

[0197] S902, decoding the position embedding information of the coded visual block and the non-critical block in the target image through a visual decoder to obtain a decoded visual block.

[0198] Specifically, this embodiment can input the position embedding information of the visual coding block and the non-critical block in the target image into the visual decoder together. The visual decoder can not only decode the coded visual block to obtain the decoded visual block corresponding to the key block, but also can decode the decoded visual block corresponding to the non-critical block in the target image according to the position embedding information and the coded visual block of the non-critical block in the target image.

[0199] For example, Fig.10 As shown, taking the Transformer decoder as the visual decoder used in this step as an example, Fig.10 The square filled with diagonal lines in the figure represents the encoded visual blocks (i.e., Encoded VisiblePatches) corresponding to the key blocks, and the black square marks represent the non-key blocks in the target image. Since the non-key blocks are not encoded, the non-key blocks here are mask blocks (i.e., Mask Tokens), and their encoded data can be a random value or a preset fixed value. This embodiment can be Fig.10The coded visual blocks and mask blocks in the image are embedded with their position information in the target image and then input into the first normalization network of the Transformer decoder. Based on the first normalization network, each coded block (i.e., each coded visual block and mask block) is first normalized to obtain the first normalization result of each coded block. Then, for the first normalization result of each coded block, the three matrices Q corresponding to each coded block are determined according to the following formula (5): t , K t 、V t .

[0200]

[0201] Where t is the first normalization processing result corresponding to each coding block, [W t Q ,W t K ,W t V ] is the attention weight matrix of each head in the randomly initialized multi-head self attention mechanism in the Transformer decoder, Q t , K t and V t These are the three matrices corresponding to each coding block.

[0202] The three matrices Q corresponding to each coding block t , K t and V t The input is fed into a randomly initialized Multi-head Self-Attention network, which determines the multi-head value corresponding to each encoding block based on the following formulas (6) and (7).

[0203]

[0204]

[0205] Among them, s t The encoding block t corresponds to each attention head, m Kt is the matrix K t , δ() is the function for calculating attention, and τ() is the Softmax logistic regression function. is the connection function (i.e. Concatenate function), W t e is the parameter matrix, and r is the multi-head value corresponding to the encoding block t.

[0206] The multi-head value corresponding to each coding block, the block coding data and position embedding information of the coding block are fused, and the fusion result is input into the second standardization network. The second standardization result output by the second standardization network is input into the multi-layer perception (MLP) network of the Transformer decoder to obtain the perception result of each coding block, that is, the decoding result of each coding block, that is, the decoded visual block corresponding to each coding block. The reconstructed target image can be obtained through each decoded visual block.

[0207] In this embodiment, the feature code stream received by the edge end includes not only the encoded visual block, but also the position embedding information of the non-critical block in the target image. Since the encoded visual block also includes the position embedding information of the non-critical block in the target image, the visual decoder of this embodiment uses the original position embedding information of the critical block and the non-critical block in the target image during the decoding process. It can not only ensure the accuracy of decoding the critical block, but also accurately locate other blocks around each non-critical block, so as to determine the decoded visual block of each non-critical block based on the block encoding data of other surrounding blocks, thereby improving the accuracy of each decoded visual block finally decoded.

[0208] In one embodiment, in order to ensure the accuracy of the edge decoding result, the visual decoder at the edge needs to be regularly updated and trained. Specifically, the embodiment of the present application can introduce the cloud, and complete the update training of the visual decoder at the edge through the cooperation between the cloud and the edge. Specifically, after the feature code stream is decoded in S802 and the decoded visual block is obtained, the edge can send the decoded visual block to the cloud, and the decoded visual block is used for the cloud to update the training data set. Correspondingly, the subsequent edge may receive the trained visual decoder sent by the cloud, and use the trained visual decoder to replace the local visual decoder, wherein the trained visual decoder is trained based on the updated training data set. Specifically, each time the edge obtains the decoded visual block, it will be sent to the cloud, and the cloud will update the training data set based on the received decoded visual block. When the updated training data set meets the training conditions, the cloud will use the updated training data set to train the visual decoder, and feed the trained visual decoder back to the edge. After receiving the trained visual decoder sent by the cloud, the edge will replace the local visual decoder with it to complete the update of the local visual decoder at the edge. This enables the effect of continuous updating of the visual decoder deployed in the Internet of Things.

[0209] Optionally, if the edge end performs a visual task after obtaining the decoded visual block and obtains a task processing result, in order to ensure the accuracy of the cloud's training of the visual decoder, the edge end can send the task processing result at the same time as sending the decoded visual block to the cloud, so that the cloud can determine whether to use the decoded visual block to update the training data set based on the task processing result, thereby ensuring the accuracy of the updated training data set, and thereby improving the accuracy of the visual decoder trained based on the training data set.

[0210] It should be noted that the process of how the cloud updates the training data set based on the decoded visual blocks (or decoded visual blocks and task processing results) sent by the edge, and how to continue training the visual decoder based on the updated training data set will be described in detail in subsequent embodiments and will not be repeated here.

[0211] In one embodiment, Fig.11 As shown, a visual task processing method is provided, which is applied to Figure 1 The cloud 300 in the example is used to illustrate, and the cloud 300 is mainly used to connect the user end 400 and the edge end 200 to assist in realizing visual task processing based on the visual Internet of Things. The following steps are included:

[0212] S1101, receiving a task calling request sent by a user terminal.

[0213] Among them, the task call request can be a request initiated by the user end to request the visual Internet of Things to call the machine vision algorithm to perform the machine vision task when the user has a machine vision task processing demand.

[0214] Optionally, in this embodiment, when a user has a need to process a machine vision task, the user can select the type of machine vision task to be executed on the user terminal, and the user terminal can generate a task call request according to the type of machine vision task selected by the user, and send it to the cloud of the visual Internet of Things that can execute the machine vision task. Correspondingly, the cloud of the visual Internet of Things can receive the task call request sent by the user terminal.

[0215] S1102, based on the task calling request, sending the original visual task processing request to the edge end.

[0216] Among them, the original visual task processing request is used to instruct the edge end to send a target visual task processing request to the terminal; the target visual task processing request is used to instruct the terminal to determine the key blocks from each image block according to the pooling information of each other image block obtained by dividing the target image to be encoded, and encode the key blocks to obtain a feature code stream, and the edge end determines the task processing result according to the feature code stream. It should be noted that how the edge end initiates a target visual task processing request to the terminal according to the original visual task processing request, how the terminal responds to the target visual task request and sends the visually encoded feature code stream to the edge end, and how the edge end determines the task processing result according to the feature code stream has been introduced in the above embodiment and will not be repeated here.

[0217] Optionally, the cloud sends the original task processing request to the edge based on the task call request by parsing the task call request sent by the user, determining the type of machine learning task that the user needs to call, and then determining the edge that executes the machine learning task in the visual Internet of Things based on the parsing result, and initiating the original task processing request to the corresponding edge in accordance with the communication standard of the visual Internet of Things.

[0218] In the above-mentioned visual task processing method, the cloud triggers the start of the visual task processing process by interacting with the edge; in the visual task processing process, the terminal accurately extracts the key pixel area with rich image information in the target image, namely the key block, based on the pooling information that characterizes the pixel changes, and only encodes the key block, thereby achieving the effect of selectively encoding the key pixel area in the image data. Compared with encoding all the pixels in the target image, the amount of data of the encoded feature code stream is reduced, thereby improving the efficiency of transmitting the feature code stream to the edge, thereby improving the efficiency of the entire visual task processing.

[0219] Optionally, in order to ensure the security of visual task execution, after receiving the task call request sent by the user end, the cloud can also obtain the user information associated with the user end according to the task call request; perform permission verification on the user information; and send the original visual task processing request to the edge end if the permission verification passes. Specifically, if the task call request contains user information, the user information contained in the task call request can be used as the user information associated with the user end; if the task call request does not contain user information, the information of the user end that sent the task call request can be used as the user information associated with the user end. When the user information is verified for permission, the user information associated with the user end obtained at this time can be compared with the authorized user information pre-stored in the cloud. If the user information associated with the user end belongs to the authorized user information, it is determined that the user information permission verification is passed, and the original visual task processing request can be sent to the edge end at this time. Otherwise, it is considered that the user information permission verification fails, and the subsequent operation of initiating the original visual task processing request to the edge end is not performed. In this embodiment, the cloud verifies the user information permissions of the task call request initiated by the user end and then initiates the original visual task processing request to the cloud of the visual Internet of Things, thereby preventing unauthorized users from arbitrarily triggering the visual Internet of Things to perform visual tasks, thereby improving the security of visual task execution.

[0220] In one embodiment, in order to ensure the accuracy of the encoding and decoding results, the visual decoder and / or the visual encoder need to be regularly updated and trained. Specifically, the embodiment of the present application can introduce cooperation between the cloud and the edge to complete the update training of the visual decoder and / or the visual encoder. Fig.12 As shown, after the method sends the original visual task processing request to the edge end in S1102, it also includes the following steps:

[0221] S1201, receiving a decoded visual block sent by an edge end.

[0222] Optionally, after the cloud sends the original visual task processing request to the edge, the edge will interact with the terminal to execute the original visual task processing request (the specific implementation method has been introduced in the above embodiment and will not be repeated here). During this process, the edge will generate a decoded visual block and a task processing result corresponding to the original visual task processing request. In this embodiment, the edge can only feedback the decoded visual block obtained when executing the original visual task processing request to the cloud after each execution of the original visual task processing request. It can also feedback the decoded visual block to the cloud while feeding back the task processing result. There is no limitation on this. Accordingly, the cloud will receive the decoded visual block or the decoded visual block and the task processing result fed back by the edge each time the original visual task processing request is sent to the edge.

[0223] S1202, using the decoded visual block to update the training data set.

[0224] Specifically, after receiving the decoded visual block fed back by the edge, the cloud can first reconstruct the decoded target image based on the decoded visual block, and then directly add the reconstructed target image to the training data set to achieve an update of the training data set.

[0225] Optionally, if the cloud not only obtains the decoded visual block but also obtains the task processing result in step S1201, then this step can also first determine whether the decoded visual block received this time meets the training set update condition based on the task processing result; if it does, the decoded visual block is used to update the training data set. Among them, the training set update condition can be updated when a positive sample is encountered.

[0226] Specifically, according to the task processing result, it is determined whether the target image reconstructed by the decoded visual block received this time belongs to a positive sample. If so, it is determined that the training set update condition is met, and the decoded visual block can be continued to be used to update the training data set. For example, if the visual task of this embodiment is a defect recognition task, then if the task processing result is that there is a defect, it is determined that the received decoded visual block meets the training set update condition, and the operation of using the decoded visual block to update the training data set can be performed; if the task processing result is that there is no defect, it is determined that the received decoded visual block does not meet the training set update condition, that is, the decoded visual block cannot be used to update the training data set.

[0227] S1203: When the updated training data set meets the training conditions, the visual encoder and / or the visual decoder are trained using the updated training data set.

[0228] It should be noted that in this embodiment, after the cloud sends the original visual task processing request to the edge each time, it will update the training data set based on the decoded visual block received from the edge feedback (or update the training data set when it is determined that the training set update conditions are met based on the task processing results). After each update of the training data set, it will trigger a judgment on whether the updated training data set meets the training conditions. For example, it can be judged whether the total amount of data in the updated training data set reaches a preset number. If so, it is judged that the training conditions are met. It can also be whether the time interval between the training data set and the last time the training conditions were met reaches a preset duration. If so, it is determined that the training conditions are met.

[0229] If the updated training data set meets the training conditions, the updated training data set is used as a training sample to perform supervised or unsupervised training on the visual encoder and / or visual decoder trained last time on the cloud. If the updated training data set does not meet the training conditions, this step and subsequent operations are not performed. It should be noted that if the cloud is training the visual encoder and / or visual decoder for the first time, the training object at this time can be the visual decoder and visual encoder deployed at the edge and terminal in the visual Internet of Things.

[0230] It should also be noted that in this embodiment, after the cloud uses the updated training data set to train the visual encoder and / or visual decoder, it can clear the training data set so that subsequent iterative updates can obtain a new training data set to train the visual encoder and / or visual decoder.

[0231] S1204: Send the trained visual decoder to the edge end, and / or send the trained visual encoder to the terminal.

[0232] Optionally, in this embodiment, the cloud can send the visual decoder of this training to the edge of the visual Internet of Things, and send the visual encoder of this training to the terminal of the visual Internet of Things after each training. It can also be that after each training, it is determined whether the accuracy of the visual decoders and visual encoders before and after has improved. If the accuracy has improved, it will be sent to the corresponding edge or terminal. For example, if the accuracy of the visual decoder after training is higher than that of the visual decoder before training, the visual decoder is sent to the edge. If the accuracy of the visual encoder after training is almost unchanged from the accuracy of the visual encoder before training, or the accuracy is reduced, the visual encoder after this training will not be sent to the terminal.

[0233] In this embodiment, the cloud obtains the decoded visual blocks obtained by the edge end when connecting to the user end and the edge end to assist in the processing of visual tasks based on the visual Internet of Things, and updates the training data set based on the decoded visual blocks. When the training data set meets the training conditions, the visual encoder and / or visual decoder are trained, and the trained visual encoder and / or visual decoder are fed back to the corresponding terminal and / or edge end. In the process of the visual Internet of Things performing visual tasks, the visual encoder deployed in the terminal of the visual Internet of Things and the visual decoder deployed at the edge end are continuously updated, which provides a guarantee for the accurate execution of visual tasks by the visual Internet of Things.

[0234] In one embodiment, the present application provides a coding and decoding system based on combined pooling Transformer, which is essentially a coding and decoding system based on cloud-edge collaboration. Fig.13As shown, the system includes a cloud, an edge, and a terminal.

[0235] Among them, the cloud is at the application layer of the system and can be used to connect to the user end. For example, the cloud can receive the user's task call request through the user end; further, the cloud can also be used to connect to the edge end to trigger the edge end to start the processing flow of the visual task. For example, the cloud can send the original visual task processing request to the edge end, so that the edge end can trigger the terminal to execute the encoding process in the visual task processing process based on the original visual task processing request. In addition, the cloud can also regularly update the Transformer-based visual decoder integrated in the edge end. Furthermore, the cloud can also connect to the terminal, for example, the cloud can regularly update the Transformer-based visual encoder integrated in the terminal.

[0236] The edge is located in the service and application support layer of the system, and is mainly used to perform the decoding process and specific machine vision tasks (such as target detection / recognition) in the visual task processing process. Optionally, the edge can be used to connect to the cloud, for example, to obtain the original visual task processing request sent by the cloud, to send to the cloud the decoded visual block obtained by decoding the feature code stream sent by the terminal, and to feedback the task processing results of the machine vision task to the cloud.

[0237] Furthermore, the edge end can also be connected to a terminal located at the device layer of the system. For example, a target visual task processing request can be sent to the terminal to trigger the terminal to execute the encoding process in the entire visual task processing process.

[0238] exist Fig.13 Based on the system shown in FIG. 1 , the embodiment of the present application provides a process for visual task processing to be realized by mutual interaction between the user end, the cloud end, the edge end and the terminal. Fig.14 As shown, the method includes the following steps.

[0239] S1401, the user terminal sends a task call request to the cloud.

[0240] S1402, the cloud receives a task calling request sent by the user.

[0241] S1403, the cloud obtains the user information associated with the user end according to the task call request, and performs permission verification on the user information.

[0242] S1404: When the authority verification is passed, the cloud sends the original visual task processing request to the edge.

[0243] S1405, the edge receives the original visual task processing request sent by the cloud.

[0244] S1406: The edge sends a target visual task processing request to the terminal based on the original visual task processing request.

[0245] S1407, the terminal receives the target visual task processing request sent by the edge end.

[0246] S1408, the terminal obtains the target image to be encoded corresponding to the target visual task processing request.

[0247] Optionally, the terminal may control the image acquisition device to acquire the target image to be encoded corresponding to the target visual task processing request.

[0248] S1409: The terminal divides the target image into at least two image blocks, and determines a combined pooling value of each image block according to a maximum pooling value and an average pooling value of each image block.

[0249] Optionally, the terminal can determine the maximum pooling value and the average pooling value of each image block based on the pixel value of each pixel point contained in each image block; the terminal uses a preset pooling ratio to perform weighted summation on the maximum pooling value and the average pooling value of each image block to obtain a combined pooling value for each image block.

[0250] S1410, the terminal determines the number of non-critical blocks according to a preset compression ratio and the number of image blocks, and selects a critical block from at least two image blocks according to the number of non-critical blocks and a combined pooling value of each image block.

[0251] S1411, the terminal encodes the key blocks to obtain coded visual blocks, embeds information according to the positions of the coded visual blocks and non-key blocks in the target image, and generates a feature code stream.

[0252] Optionally, the terminal can perform linear projection on the key block to obtain the block embedding information of the key block, and encode the block embedding information of the key block and the position embedding information of the key block in the feature map through a visual encoder to obtain the encoded visual block corresponding to the key block. Further, a feature code stream is generated based on the position embedding information of the encoded visual block and the non-key block in the feature map.

[0253] S1412: The terminal sends the feature code stream to the edge end.

[0254] S1413: The edge end receives the feature code stream sent by the terminal.

[0255] S1414, the edge end obtains the position embedding information of the coded visual block and the non-critical block in the target image from the feature code stream.

[0256] S1415, the edge end decodes the position embedding information of the encoded visual block and the non-critical block in the target image through the visual decoder to obtain the decoded visual block, and performs the visual task according to the decoded visual block to obtain the task processing result, and feeds back the task processing result to the user end.

[0257] Optionally, the edge terminal can not only feed back the task processing result to the user terminal, but also perform statistical analysis of the visual task and display the analysis result, etc. There is no limitation on this.

[0258] S1416, the edge sends the decoded visual block to the cloud.

[0259] S1417, the cloud receives the decoded visual block sent by the edge.

[0260] S1418, the cloud uses the decoding vision block to update the training data set, and when the updated training data set meets the training conditions, the updated training data set is used to train the visual encoder and the visual decoder.

[0261] S1419, the cloud sends the trained visual decoder to the edge, and sends the trained visual encoder to the terminal.

[0262] S1420, the terminal receives the trained visual encoder sent by the cloud, and uses the trained visual encoder to replace the local visual encoder.

[0263] S1421, the edge receives the trained visual decoder sent by the cloud, and uses the trained visual decoder to replace the local visual decoder.

[0264] It should be understood that although Figure 1-14 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1-14 At least part of the steps may include multiple steps or multiple stages. These steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed in turn or alternately with other steps or at least part of the steps or stages in other steps.

[0265] In one embodiment, Fig.15 As shown, a visual task processing device 1 is provided, comprising: an image division module 10, a key block determination module 11, an encoding module 12 and a code stream sending module 13, wherein:

[0266] An image division module 10, used for dividing a target image to be encoded into at least two image blocks;

[0267] A key block determination module 11, configured to determine a key block from at least two image blocks according to pooling information of each image block;

[0268] The encoding module 12 is used to encode the key blocks to obtain a feature code stream;

[0269] The code stream sending module 13 is used to send the feature code stream to the edge end.

[0270] In one embodiment, the key block determination module 11 includes: a combined pooling determination unit and a key block determination unit, wherein:

[0271] A combined pooling determination unit, used to determine a combined pooling value of each image block according to the pixel value of each pixel point contained in each image block;

[0272] The key block determination unit is used to determine the key block from at least two image blocks according to the combined pooling value of each image block.

[0273] In one embodiment, the combined pooling determination unit may be specifically used for:

[0274] For each image block, the maximum pooling value and the average pooling value of the image block are determined according to the pixel value of each pixel point contained in the image block; using the preset pooling ratio, the maximum pooling value and the average pooling value of the image block are weighted summed to obtain the combined pooling value of the image block.

[0275] In one embodiment, the key block determination unit includes: a quantity determination subunit and a key block determination subunit, wherein:

[0276] A quantity determination subunit, used for determining the number of non-critical blocks according to a preset compression ratio and the number of image blocks;

[0277] The key block determination subunit is used to determine the key block from at least two image blocks according to the number of non-key blocks and the combined pooling value of each image block.

[0278] In one embodiment, the key block determination subunit may be specifically used for:

[0279] According to the combined pooling value of each image block, at least two image blocks are sorted; according to the number of non-critical blocks and the sorting result, non-critical blocks are selected from the at least two image blocks; and image blocks other than the non-critical blocks in the at least two image blocks are used as critical blocks.

[0280] In one embodiment, the encoding module 12 includes: an encoding unit and a code stream generating unit, wherein:

[0281] A coding unit, used for coding the key blocks to obtain coded visual blocks;

[0282] The code stream generating unit is used to generate a feature code stream according to the position embedding information of the coded visual block and the non-critical block in the target image.

[0283] In one embodiment, the encoding unit may be specifically used for:

[0284] Linearly project the key block to obtain the block embedding information of the key block; encode the block embedding information of the key block and the position embedding information of the key block in the target image through the visual encoder to obtain the encoded visual block corresponding to the key block.

[0285] In one embodiment, the visual task processing device 1 further includes: an encoder update module, which can be used to:

[0286] Receive the trained visual encoder sent by the cloud; wherein the trained visual encoder is trained by the cloud based on the training data set updated by the received decoded visual block; the decoded visual block is obtained by the edge end based on the feature code stream decoding; the trained visual encoder is used to replace the local visual encoder.

[0287] In one embodiment, the visual task processing device 1 further includes: an image acquisition module, which is used to:

[0288] Receive a target visual task processing request sent by the edge end; and obtain a target image to be encoded based on the target visual task processing request.

[0289] In one embodiment, the target visual task processing request is generated by the edge based on the original visual task processing request sent by the cloud; the original visual task processing request is generated by the cloud based on the task call request sent by the user.

[0290] In one embodiment, Fig.16 As shown, a visual task processing device 2 is provided, comprising: a code stream receiving module 20 and a decoding module 21, wherein:

[0291] The code stream receiving module 20 is used to receive a characteristic code stream sent by the terminal; wherein the characteristic code stream is obtained by encoding the key block by the terminal, and the key block is the pooling information of each image block obtained by the terminal according to dividing the target image to be encoded, and is determined from each image block;

[0292] The decoding module 21 is used to decode the feature code stream to obtain a decoded visual block.

[0293] In one embodiment, the feature code stream is generated by the terminal according to the encoded visual block obtained by encoding the key block and the position embedding information of the non-key block in the target image;

[0294] Accordingly, the decoding module 21 can be specifically used for:

[0295] The position embedding information of the coded visual block and the non-critical block in the target image is obtained from the feature code stream; the position embedding information of the coded visual block and the non-critical block in the target image is decoded by a visual decoder to obtain a decoded visual block.

[0296] In one embodiment, the visual task processing device 2 may further include: a decoder updating module, which may be used to:

[0297] Sending a decoded visual block to the cloud; wherein the decoded visual block is used for the cloud to update a training data set; receiving a trained visual decoder sent by the cloud; wherein the trained visual decoder is trained based on the updated training data set; and using the trained visual decoder to replace a local visual decoder.

[0298] In one embodiment, the visual task processing device 2 may further include:

[0299] An original receiving module, used to receive an original visual task processing request sent by the cloud; wherein the original visual task processing request is generated by the cloud based on the task call request sent by the user end;

[0300] A target sending module is used to send a target visual task processing request to the terminal based on the original visual task processing request; wherein the target visual task processing request is used to request the terminal to feedback a feature code stream;

[0301] The task processing module is used to perform visual tasks according to the decoded visual blocks and obtain task processing results;

[0302] The result feedback module is used to feedback the task processing results to the user end.

[0303] In one embodiment, Fig.17 As shown, a visual task processing device 3 is provided, comprising: a request receiving module 30 and a request sending module 31, wherein:

[0304] The request receiving module 30 is used to receive the task calling request sent by the user end;

[0305] A request sending module 31, used to send an original visual task processing request to the edge end based on the task call request;

[0306] Among them, the original visual task processing request is used to instruct the edge end to send a target visual task processing request to the terminal; the target visual task processing request is used to instruct the terminal to determine the key blocks from each image block according to the pooling information of each image block obtained by dividing the target image to be encoded, and encode the key blocks to obtain a feature code stream, and the edge end determines the task processing result according to the feature code stream.

[0307] In one embodiment, the request sending module 31 may be specifically used for:

[0308] According to the task call request, obtain the user information associated with the user end; verify the authority of the user information; if the authority verification passes, send the original visual task processing request to the edge end.

[0309] In one embodiment, the visual task processing device 3 further includes: a training module, which can be used to:

[0310] Receive a decoded visual block sent by an edge end; use the decoded visual block to update a training data set; when the updated training data set meets the training conditions, use the updated training data set to train the visual encoder and / or visual decoder; send the trained visual decoder to the edge end, and / or send the trained visual encoder to the terminal.

[0311] For the specific definition of the visual task processing device, please refer to the definition of the visual task processing method above, which will not be repeated here. Each module in the above-mentioned visual task processing device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0312] Fig.18A schematic diagram of the structure of an edge / terminal device provided for an embodiment of the present application. The edge / terminal device 4 may include a receiver 41, a memory 42, a processor 43, at least one communication bus 44 and a transmitter 45. The communication bus 44 is used to realize the communication connection between the components. The memory 42 may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory. Various programs may be stored in the memory 42 to complete various processing functions and implement the method steps of this embodiment. In this embodiment, the transmitter 45 may be a radio frequency processing module or a baseband processing module in an access network device, and the receiver 41 may also be a radio frequency processing module or a baseband processing module in an access network device. The transmitter 45 and the receiver 41 can be integrated together to realize a transceiver, and the transmitter 45 and the receiver 41 can be coupled to the processor 43, which can realize the receiving or sending action under the instruction or control of the processor 43.

[0313] In this embodiment, if Fig.18 When it is a terminal device, the processor 43 is used to divide the target image to be encoded into at least two image blocks; determine the key block from at least two image blocks according to the pooling information of each image block; encode the key block to obtain a feature code stream; the transmitter 45 is used to send the feature code stream to the edge end.

[0314] In this embodiment, if Fig.18 When it is an edge device, the receiver 41 is used to receive the feature code stream sent by the terminal; wherein the feature code stream is obtained by the terminal encoding the key block, and the key block is the pooling information of each image block obtained by the terminal according to the division of the target image to be encoded, and is determined from each image block; the processor 43 is used to decode the feature code stream to obtain a decoded visual block.

[0315] In one embodiment, a cloud device is provided. Fig.19 . Fig.19 It is a structural diagram of a cloud device provided by an embodiment of the present invention. Fig.19 The cloud device 5 shown includes: at least one processor 51, a memory 52, at least one network interface 54 and a user interface 53. The various components in the cloud device 5 are coupled together via a bus system 55. It is understood that the bus system 55 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 55 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 55 is not shown in FIG. Fig.19 Various buses are labeled as bus system 55. In addition, in the embodiment of the present invention, a transceiver 56 is also included. The transceiver can be a plurality of components, that is, including a transmitter and a receiver, and is a unit for communicating with various other devices on a transmission medium.

[0316] The user interface 53 may include a display, a keyboard, or a pointing device (eg, a mouse, a trackball, a touch pad, or a touch screen).

[0317] It can be understood that the memory 52 in the embodiment of the present invention can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (DRRAM). The memory 52 of the system and method described in the embodiments of the present invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0318] In some implementations, the memory 52 stores the following elements, executable modules or data structures, or a subset thereof, or an extended set thereof: an operating system 521 and application programs 522 .

[0319] The operating system 521 includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic services and process hardware-based tasks. The application 522 includes various application programs, such as a media player (MediaPlayer), a browser (Browser), etc., which are used to implement various application services. The program for implementing the method of the embodiment of the present invention can be included in the application 522.

[0320] In an embodiment of the present invention, by calling a program or instruction stored in the memory 52, specifically, it can be a program or instruction stored in the application 522, wherein the receiver is used to receive a task call request sent by the user end; the transmitter is used to send an original visual task processing request to the edge end based on the task call request; wherein the original visual task processing request is used to instruct the edge end to send a target visual task processing request to the terminal; the target visual task processing request is used to instruct the terminal to determine the key blocks from each image block according to the pooling information of each image block obtained by dividing the target image to be encoded, and encode the key blocks to obtain a feature code stream, and the edge end determines the task processing result according to the feature code stream.

[0321] It is to be understood that the embodiments described in the embodiments of the present invention may be implemented by hardware, software, firmware, middleware, microcode or a combination thereof. For hardware implementation, the processing unit may be implemented in one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSP devices, DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), general purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described in the present application or a combination thereof.

Claims

1. A visual task processing method, characterized in that: Applied to a terminal, the method comprises: Dividing a target image to be encoded into at least two image blocks; Determining a key block from the at least two image blocks according to the pooling information of each image block; The key blocks are encoded to obtain a feature code stream, and the feature code stream is sent to the edge end.

2. The method according to claim 1, characterized in that: The determining the key block from the at least two image blocks according to the pooling information of each image block comprises: Determine a combined pooling value of each image block according to the pixel value of each pixel point contained in each image block; A key block is determined from the at least two image blocks according to the combined pooled value of each image block.

3. The method according to claim 2, characterized in that The step of determining the combined pooling value of each image block according to the pixel value of each pixel point included in each image block includes: For each image block, determine the maximum pooling value and the average pooling value of the image block according to the pixel value of each pixel point contained in the image block; Using the preset pooling ratio, the maximum pooling value and the average pooling value of the image block are weighted summed to obtain the combined pooling value of the image block.

4. The method according to claim 3, characterized in that The determining the key block from the at least two image blocks according to the combined pooling value of each image block comprises: Determine the number of non-critical blocks according to a preset compression ratio and the number of image blocks; A key block is determined from the at least two image blocks according to the number of non-key blocks and the combined pooling value of each image block.

5. The method according to claim 4, characterized in that The determining of the key block from the at least two image blocks according to the number of non-key blocks and the combined pooling value of each image block comprises: sorting the at least two image blocks according to the combined pooling value of each image block; Selecting a non-key block from the at least two image blocks according to the number of the non-key blocks and the sorting result; The image blocks other than the non-key blocks in the at least two image blocks are used as key blocks.

6. The method according to claim 5, characterized in that The step of encoding the key blocks to obtain a feature code stream includes: Encoding the key blocks to obtain encoded visual blocks; Information is embedded according to the positions of the coded visual blocks and the non-critical blocks in the target image to generate a feature code stream.

7. The method according to claim 6, characterized in that The step of encoding the key blocks to obtain encoded visual blocks includes: Performing linear projection on the key block to obtain block embedding information of the key block; The block embedding information of the key block and the position embedding information of the key block in the target image are encoded by a visual encoder to obtain an encoded visual block corresponding to the key block.

8. The method according to claim 7, characterized in that The method further comprises: Receiving a trained visual encoder sent by the cloud; wherein the trained visual encoder is obtained by training the cloud based on a training data set updated by the received decoded visual block; and the decoded visual block is obtained by decoding the edge end based on the feature code stream; Use the trained visual encoder to replace the local visual encoder.

9. The method according to claim 1, characterized in that: The method further comprises: Receive the target visual task processing request sent by the edge end; Based on the target visual task processing request, a target image to be encoded is obtained.

10. The method according to claim 9, characterized in that The target visual task processing request is generated by the edge based on the original visual task processing request sent by the cloud; the original visual task processing request is generated by the cloud based on the task call request sent by the user.

11. A visual task processing method, characterized in that: Applied to the edge end, the method includes: A feature code stream sent by a receiving terminal; wherein the feature code stream is obtained by encoding a key block by the terminal, and the key block is determined from each image block by the terminal according to pooling information of each image block obtained by dividing a target image to be encoded; The feature code stream is decoded to obtain a decoded visual block.

12. The method according to claim 11, characterized in that The feature code stream is generated by the terminal according to the coded visual block obtained by encoding the key block and the position embedding information of the non-key block in the target image; Correspondingly, decoding the feature code stream to obtain a decoded visual block includes: Acquire the position embedding information of the coded visual block and the non-critical block in the target image from the feature code stream; The position embedding information of the coded visual block and the non-critical sub-block in the target image is decoded by a visual decoder to obtain a decoded visual block.

13. The method according to claim 12, characterized in that The method further comprises: Sending the decoded visual block to the cloud; wherein the decoded visual block is used for the cloud to update the training data set; Receiving a trained visual decoder sent by the cloud; wherein the trained visual decoder is trained based on an updated training data set; Use the trained visual decoder to replace the local visual decoder.

14. The method according to any one of claims 11 to 13, characterized in that The method further comprises: Receiving an original visual task processing request sent by a cloud; wherein the original visual task processing request is generated by the cloud based on a task call request sent by a user terminal; Based on the original visual task processing request, sending a target visual task processing request to the terminal; wherein the target visual task processing request is used to request the terminal to feedback the feature code stream; Accordingly, the method further includes: According to the decoded visual block, a visual task is performed to obtain a task processing result; Feedback the task processing result to the user terminal.

15. A visual task processing method, characterized in that: Applied to the cloud, the method includes: Receive the task call request sent by the user; Based on the task calling request, sending an original visual task processing request to the edge end; Among them, the original visual task processing request is used to instruct the edge end to send a target visual task processing request to the terminal; the target visual task processing request is used to instruct the terminal to determine the key blocks from each image block according to the pooling information of each image block obtained by dividing the target image to be encoded, and encode the key blocks to obtain a feature code stream, and the edge end determines the task processing result according to the feature code stream.

16. The method according to claim 15, characterized in that The sending the original visual task processing request to the edge end based on the task calling request includes: Acquire user information associated with the user terminal according to the task call request; Performing authority verification on the user information; If the authority verification is passed, the original visual task processing request is sent to the edge.

17. The method according to claim 15, characterized in that The method further comprises: Receiving a decoded visual block sent by the edge end; Using the decoded visual block, updating the training data set; When the updated training data set meets the training conditions, the visual encoder and / or the visual decoder are trained using the updated training data set; The trained visual decoder is sent to the edge end, and / or the trained visual encoder is sent to the terminal.

18. A visual task processing device, characterized in that: Configured in a terminal, the device includes: An image division module, used for dividing a target image to be encoded into at least two image blocks; A key block determination module, configured to determine a key block from the at least two image blocks according to the pooling information of each image block; An encoding module, used for encoding the key blocks to obtain a feature code stream; The code stream sending module is used to send the characteristic code stream to the edge end.

19. A visual task processing device, characterized in that: Configured in the edge end, the device includes: A code stream receiving module, used for receiving a characteristic code stream sent by a terminal; wherein the characteristic code stream is obtained by encoding a key block by the terminal, and the key block is determined from each image block by the terminal according to pooling information of each image block obtained by dividing a target image to be encoded; The decoding module is used to decode the feature code stream to obtain a decoded visual block.

20. A visual task processing device, characterized in that: Configured in the cloud, the device includes: A request receiving module, used to receive a task calling request sent by a user terminal; A request sending module, used for sending an original visual task processing request to the edge end based on the task call request; Among them, the original visual task processing request is used to instruct the edge end to send a target visual task processing request to the terminal; the target visual task processing request is used to instruct the terminal to determine the key blocks from each image block according to the pooling information of each image block obtained by dividing the target image to be encoded, and encode the key blocks to obtain a feature code stream, and the edge end determines the task processing result according to the feature code stream.

21. A terminal device, characterized in that: include: a first processor and a first transmitter; The first processor is used to divide the target image to be encoded into at least two image blocks, and determine a key block from the at least two image blocks according to pooling information of each image block, and encode the key block to obtain a feature code stream; The first transmitter is used to send the feature code stream to the edge end.

22. An edge device, characterized in that: include: a first receiver and a second processor; The first receiver is used to receive a feature code stream sent by a terminal; wherein the feature code stream is obtained by the terminal encoding a key block, and the key block is determined from each image block by the terminal according to pooling information of each image block obtained by dividing a target image to be encoded; The second processor is used to decode the feature code stream to obtain a decoded visual block.

23. A cloud device, characterized in that: include: a second receiver and a second transmitter; The second receiver is used to receive a task calling request sent by a user terminal; The second transmitter is used to send an original visual task processing request to the edge end based on the task calling request; Among them, the original visual task processing request is used to instruct the edge end to send a target visual task processing request to the terminal; the target visual task processing request is used to instruct the terminal to determine the key blocks from each image block according to the pooling information of each image block obtained by dividing the target image to be encoded, and encode the key blocks to obtain a feature code stream, and the edge end determines the task processing result according to the feature code stream.

24. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 17 are implemented.

Images