Visual task processing method and device, equipment and storage medium

Abstract

The invention relates to a visual task processing method and device, equipment and a storage medium. When the method is applied to the terminal, the method comprises the following steps: performing feature extraction on a to-be-coded target image to obtain a feature map; determining a key feature region in the feature map according to the angular point feature of each feature point in the feature map; and encoding the key feature region to obtain a feature code stream, and sending the feature code stream to an edge end. By adopting the method, the visual task can be quickly responded.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a visual task processing method, device, equipment and storage medium, which can be applied to the field of machine vision technology. Background Art

[0002] The visual Internet of Things is an upgraded version of the traditional Internet of Things. It uses various visual sensors, including cameras, mobile phones, surveillance cameras, etc., to collect images of objects, people, vehicles, etc., and uses image processing algorithms and machine vision tasks to calculate and process these data to obtain valuable information from the images. The information is then transmitted to the application end (such as the edge end) through the network, and after post-processing and intelligent algorithm analysis at the application end, communication and information exchange are achieved.

[0003] With the rapid development of 5G, big data, and artificial intelligence, visual tasks in the field of visual IoT are becoming increasingly popular. Common visual tasks include image classification, video object detection, and image segmentation. The processing of visual tasks in the field of visual IoT usually involves image encoding and decoding technology. However, due to the limitation of the communication bandwidth of the visual IoT, the existing image encoding and decoding technology cannot meet the needs of fast response to visual tasks. Summary of the invention

[0004] The embodiments of the present application provide a visual task processing method, apparatus, device and storage medium, which can achieve rapid response to visual tasks.

[0005] In a first aspect, an embodiment of the present application provides a visual task processing method for a terminal, the method comprising:

[0006] Perform feature extraction on the target image to be encoded to obtain a feature map;

[0007] According to the corner point features of each feature point in the feature map, determine the key feature area in the feature map;

[0008] The key feature area is encoded to obtain a feature code stream, and the feature code stream is sent to the edge end.

[0009] In one embodiment, determining a key feature area in a feature map according to corner features of each feature point in the feature map includes:

[0010] Divide the feature map into at least two feature blocks;

[0011] According to the corner feature of each feature point in each feature block, a key block is determined from at least two feature blocks, and the key block is used as a key feature area.

[0012] In one embodiment, determining a key block from at least two feature blocks according to the corner feature of each feature point in each feature block includes:

[0013] Using the bilinear interpolation algorithm, according to the corner point features of each feature point in each feature block, the bilinear average interpolation of each feature block is determined;

[0014] A key block is determined from at least two feature blocks according to bilinear average interpolation of each feature block.

[0015] In one embodiment, a bilinear interpolation algorithm is used to determine the bilinear average interpolation of each feature block according to the corner point features of each feature point in each feature block, including:

[0016] Using a bilinear interpolation algorithm, according to the corner point features of each feature point in each feature block, the bilinear interpolation of each feature point in each feature block is determined;

[0017] According to the bilinear interpolation of each feature point in each feature block, the bilinear average interpolation of each feature block is determined.

[0018] In one embodiment, the corner point features include: corner point positions and corner point values.

[0019] In one embodiment, determining a key block from at least two feature blocks according to bilinear average interpolation of each feature block includes:

[0020] Determine the number of non-critical blocks according to a preset compression ratio and the number of feature blocks;

[0021] A key block is determined from at least two feature blocks according to the number of non-key blocks and bilinear average interpolation of each feature block.

[0022] In one embodiment, determining a key block from at least two feature blocks according to the number of non-key blocks and bilinear average interpolation of each feature block includes:

[0023] Sort at least two feature blocks according to the bilinear average interpolation of each feature block;

[0024] According to the number and ranking results of the non-key blocks, a non-key block is selected from at least two feature blocks;

[0025] The feature blocks other than the non-key blocks in at least two feature blocks are used as key blocks.

[0026] In one embodiment, encoding the key feature area to obtain a feature code stream includes:

[0027] Encode the key blocks in the key feature area to obtain the encoded visual blocks;

[0028] The feature code stream is generated based on the position embedding information of the coded visual blocks and non-critical blocks in the feature map.

[0029] In one embodiment, encoding the key blocks in the key feature area to obtain the encoded visual blocks includes:

[0030] Perform linear projection on the key blocks in the key feature area to obtain the block embedding information of the key blocks;

[0031] Through the visual encoder, the block embedding information of the key block and the position embedding information of the key block in the feature map are encoded to obtain the encoded visual block corresponding to the key block.

[0032] In one embodiment, the method further includes: receiving a trained visual encoder sent by the cloud; wherein the trained visual encoder is obtained by training the cloud based on a training data set updated by the received decoded visual block; and the decoded visual block is obtained by decoding the edge end based on the feature code stream;

[0033] Use the trained visual encoder to replace the local visual encoder.

[0034] In one of the embodiments, the method further includes: receiving a target visual task processing request sent by an edge end;

[0035] The image acquisition device is controlled to acquire the target image to be encoded corresponding to the target visual task processing request.

[0036] In one of the embodiments, the target visual task processing request is generated by the edge based on the original visual task processing request sent by the cloud; the original visual task processing request is generated by the cloud based on the task call request sent by the user.

[0037] In a second aspect, an embodiment of the present application provides a visual task processing method for an edge terminal, the method comprising:

[0038] Acquire a feature code stream sent by the terminal; wherein the feature code stream is obtained by encoding a key feature area by the terminal, and the key feature area is determined by the terminal according to corner point features of each feature point in a feature map of a target image to be encoded;

[0039] Decode the feature code stream to obtain a decoded visual block.

[0040] In one of the embodiments, the feature code stream is generated by the terminal according to the encoded visual block obtained by encoding the key block in the key feature area, and the position embedding information of the non-key block in the feature map;

[0041] Correspondingly, the feature code stream is decoded to obtain a decoded visual block, including:

[0042] Obtain the position embedding information of the coded visual block and the non-critical block in the feature map from the feature code stream;

[0043] The position embedding information of the coded visual block and the non-critical block in the feature map is decoded by the visual decoder to obtain the decoded visual block.

[0044] In one embodiment, the method further comprises:

[0045] Sending decoded visual blocks to the cloud; wherein the decoded visual blocks are used for the cloud to update the training data set;

[0046] Receiving a trained visual decoder sent by the cloud; wherein the trained visual decoder is trained based on an updated training data set;

[0047] Use the trained visual decoder to replace the local visual decoder.

[0048] In one embodiment, the method further includes: receiving an original visual task processing request sent by the cloud; wherein the original visual task processing request is generated by the cloud based on the task call request sent by the user end;

[0049] Based on the original visual task processing request, a target visual task processing request is sent to the terminal; wherein the target visual task processing request is used to request the terminal to feedback a feature code stream;

[0050] Accordingly, the method further includes:

[0051] According to the decoded visual block, the visual task is performed to obtain the task processing result;

[0052] Feedback task processing results to the user.

[0053] In a third aspect, an embodiment of the present application provides a visual task processing method for use in the cloud, the method comprising:

[0054] Receive the task call request sent by the user;

[0055] Based on the task call request, an original visual task processing request is sent to the edge end; wherein, the original visual task processing request is used to instruct the edge end to send a target visual task processing request to the terminal; the target visual task processing request is used to instruct the terminal to encode the key feature area in the feature map according to the corner point features of each feature point in the feature map of the target image to be encoded to obtain a feature code stream, and the edge end determines the task processing result according to the feature code stream.

[0056] In one embodiment, based on the task call request, sending an original visual task processing request to the edge end includes:

[0057] According to the task call request, obtain the user information associated with the user end;

[0058] Verify the permissions of user information;

[0059] If the authority verification is passed, the original visual task processing request is sent to the edge.

[0060] In one embodiment, the method further comprises:

[0061] Receive the decoded visual blocks sent by the edge end;

[0062] Adopt decoded vision blocks to update the training dataset;

[0063] When the updated training data set meets the training conditions, the visual encoder and / or the visual decoder are trained using the updated training data set;

[0064] The trained visual decoder is sent to the edge end, and / or the trained visual encoder is sent to the terminal.

[0065] In a fourth aspect, an embodiment of the present application provides a visual task processing device, which is configured in a terminal, and the device includes:

[0066] A feature extraction module is used to extract features of the target image to be encoded to obtain a feature map;

[0067] A region determination module, used to determine the key feature region in the feature map according to the corner point features of each feature point in the feature map;

[0068] The encoding and transmission module is used to encode the key feature area to obtain a feature code stream and send the feature code stream to the edge end.

[0069] In a fifth aspect, an embodiment of the present application provides a visual task processing device, which is configured at an edge end, and the device includes:

[0070] A code stream acquisition module is used to acquire a characteristic code stream sent by a terminal; wherein the characteristic code stream is obtained by encoding a key characteristic area by the terminal, and the key characteristic area is determined by the terminal according to the corner point features of each characteristic point in a characteristic map of a target image to be encoded;

[0071] The decoding module is used to decode the feature code stream to obtain a decoded visual block.

[0072] In a sixth aspect, an embodiment of the present application provides a visual task processing device, which is configured in the cloud, and the device includes:

[0073] A request receiving module, used to receive a task calling request sent by a user terminal;

[0074] The request sending module is used to send the original visual task processing request to the edge end based on the task call request; the original visual task processing request is used to instruct the edge end to send the target visual task processing request to the terminal; the target visual task processing request is used to instruct the terminal to encode the key feature area in the feature map according to the corner point features of each feature point in the feature map of the target image to be encoded to obtain a feature code stream, and the edge end determines the task processing result according to the feature code stream.

[0075] In a seventh aspect, an embodiment of the present application provides a terminal device, including: a first processor and a first transmitter;

[0076] The first processor is used to extract features of the target image to be encoded to obtain a feature map; determine the key feature area in the feature map according to the corner point features of each feature point in the feature map; encode the key feature area to obtain a feature code stream;

[0077] The first transmitter is used to send the feature code stream to the edge end.

[0078] In an eighth aspect, an embodiment of the present application provides an edge device, including: a first receiver and a second processor;

[0079] A first receiver is used to obtain a feature code stream sent by a terminal; wherein the feature code stream is obtained by encoding a key feature area by the terminal, and the key feature area is determined by the terminal according to corner point features of each feature point in a feature map of a target image to be encoded;

[0080] The second processor is used to decode the feature code stream to obtain a decoded visual block.

[0081] In a ninth aspect, an embodiment of the present application provides a cloud device, including a second receiver and a second transmitter;

[0082] A second receiver, used to receive a task calling request sent by a user terminal;

[0083] The second transmitter is used to send an original visual task processing request to the edge end based on the task call request; wherein the original visual task processing request is used to instruct the edge end to send a target visual task processing request to the terminal; the target visual task processing request is used to instruct the terminal to encode the key feature area in the feature map according to the corner point features of each feature point in the feature map of the target image to be encoded to obtain a feature code stream, and the edge end determines the task processing result according to the feature code stream.

[0084] In a tenth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the following steps are implemented:

[0085] Perform feature extraction on the target image to be encoded to obtain a feature map;

[0086] According to the corner point features of each feature point in the feature map, determine the key feature area in the feature map;

[0087] The key feature area is encoded to obtain a feature code stream, and the feature code stream is sent to the edge end.

[0088] In an eleventh aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the following steps are implemented:

[0089] Acquire a feature code stream sent by the terminal; wherein the feature code stream is obtained by encoding a key feature area by the terminal, and the key feature area is determined by the terminal according to corner point features of each feature point in a feature map of a target image to be encoded;

[0090] Decode the feature code stream to obtain a decoded visual block.

[0091] In a twelfth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the following steps are implemented:

[0092] Receive the task call request sent by the user;

[0093] Based on the task call request, an original visual task processing request is sent to the edge end; wherein, the original visual task processing request is used to instruct the edge end to send a target visual task processing request to the terminal; the target visual task processing request is used to instruct the terminal to encode the key feature area in the feature map according to the corner point features of each feature point in the feature map of the target image to be encoded to obtain a feature code stream, and the edge end determines the task processing result according to the feature code stream.

[0094] The above-mentioned visual task processing method, device, equipment and storage medium determine the key feature area in the feature map of the target image through the corner point features of each feature point in the feature map, and only encode the determined key feature area to obtain a feature code stream, and then send the feature code stream to the edge end to execute the subsequent process of visual task processing. The above-mentioned scheme, by introducing the corner point features for the feature points in the feature map, accurately extracts the key feature area with obvious features in the feature map, and only encodes the key feature area, thereby achieving the effect of selectively encoding the key feature area in the feature map. Compared with encoding all feature points in the feature map of the target image, the amount of data in the encoded feature code stream is reduced, thereby improving the efficiency of transmitting the feature code stream to the edge end, thereby improving the efficiency of the entire visual task response. BRIEF DESCRIPTION OF THE DRAWINGS

[0095] Figure 1 is an application environment diagram of a visual task processing method in one embodiment;

[0096] Figure 2 is a flowchart of a visual task processing method in one embodiment;

[0097] Figure 3 A schematic diagram of a process for determining a key feature area in a feature map in one embodiment;

[0098] Figure 4 A schematic diagram of a process of dividing a feature map into blocks in one embodiment;

[0099] Figure 5 A schematic diagram of a feature block in an embodiment;

[0100] Figure 6 A schematic diagram of a process for determining key blocks in one embodiment;

[0101] Figure 7 A schematic diagram of a process of generating a feature code stream in one embodiment;

[0102] Figure 8 is a schematic diagram of the principle of visual encoding in one embodiment;

[0103] Fig. 9 is a flow chart of a visual task processing method in another embodiment;

[0104] Fig.10 A schematic diagram of a process of decoding a feature code stream in one embodiment;

[0105] Fig.11 is a schematic diagram of the principle of visual decoding in one embodiment;

[0106] Fig.12is a flow chart of a visual task processing method in another embodiment;

[0107] Fig.13 A schematic diagram of a process of training a codec in one embodiment;

[0108] Fig.14 is an architecture diagram of a visual task processing system in one embodiment;

[0109] Fig.15 is a signaling diagram of a visual task processing method in one embodiment;

[0110] Fig.16 is a structural block diagram of a visual task processing device in one embodiment;

[0111] Fig.17 is a structural block diagram of a visual task processing device in another embodiment;

[0112] Fig.18 is a structural block diagram of a visual task processing device in another embodiment;

[0113] Fig.19 An internal structure diagram of an edge terminal / terminal device in an embodiment;

[0114] Fig. 20 1 is a diagram of the internal structure of a cloud device in one embodiment. DETAILED DESCRIPTION

[0115] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0116] Figure 1 A schematic diagram of an application scenario of a visual task processing method provided in an embodiment of the present application. The visual task may be, but is not limited to, target detection / recognition (e.g., face detection, vehicle recognition, etc.), defect detection, etc. The processing of visual tasks generally involves three stages: encoding, decoding, and detection / recognition. Figure 1 As shown, the application scenario includes a terminal 100 with an encoding function, and an edge terminal 200 with a decoding function and a detection / recognition function. The terminal 100 can interact with the edge terminal 200 through a network. For example, the terminal 100 can send a code stream obtained by encoding an image to the edge terminal 200 through the network.

[0117] In one embodiment, the application scenario may also include a cloud 300. The cloud 300 may be used to dock with the user terminal 400 to obtain the user's visual-oriented task call request; the cloud 300 may also be used to dock with the edge 200, and based on the user's task call request, trigger the edge 200 to start the processing flow of the visual task. For example, the cloud 300 triggers the edge 200 to interact with the terminal 100, so that the terminal 200 executes the process of encoding image features, and then the edge 200 decodes, detects / identifies, and performs a series of operations on the code stream obtained by the terminal 100 through the encoding technology, thereby completing the processing of the entire visual task. It is understandable that the cloud 300 can also dock with the terminal 100 to directly trigger the terminal 100 to perform the encoding process of the relevant visual task, but considering that the cloud 300 directly communicates with the terminal 100, the communication cost required is relatively large, so the embodiment of the present application uses the edge 200 to trigger the terminal 100 to perform the encoding process of the relevant visual task.

[0118] It should be noted that in the process of processing visual tasks, when traditional technology encodes the feature map, it encodes all the feature points in the feature map, resulting in a huge amount of generated code stream data, which in turn affects the efficiency of visual task processing.

[0119] Based on the above-mentioned traditional technology, an embodiment of the present application provides a visual task processing method, which improves the efficiency of visual task processing by selectively encoding feature points in a feature map.

[0120] It should be noted that the beneficial effects or technical problems solved by the embodiments of the present application are not limited to this one, but may also include other implicit or related problems. For details, please refer to the description of the following embodiments.

[0121] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0122] In one embodiment, Figure 2 As shown, a visual task processing method is provided, which is applied to Figure 1 The terminal in is used as an example to illustrate that the terminal is mainly used to perform the feature encoding part in visual task processing. It includes the following steps:

[0123] S201, extracting features from the target image to be encoded to obtain a feature map.

[0124] The so-called target image is the image that needs to be encoded. Optionally, in a visual task processing scenario, the target image can be an image corresponding to the visual task to be processed; for example, if the visual task is a face recognition task, the target image can be a face image; if the visual task is a defect detection task, the target image can be an image of an object to be detected.

[0125] Optionally, there are many ways for the terminal to obtain the target image to be encoded, which is not limited in this embodiment. For example, an image acquisition module may be integrated in the terminal, and the target image to be encoded may be obtained through the image acquisition module; the terminal may also obtain the target image to be encoded from a local image library; the terminal may also control an image acquisition device in the visual Internet of Things to acquire the target image, etc.

[0126] Optionally, in this embodiment, after acquiring the target image, the terminal may extract image features of the target image through a feature extraction network, such as a convolutional neural network, to obtain a feature map of the target image.

[0127] S202, determining a key feature area in the feature map according to the corner point features of each feature point in the feature map.

[0128] Each point in the feature map can be used as a feature point. The so-called corner point features of the feature point include: corner point position and corner point value. For each feature point, its corner point position can be the position of the four corner points of the feature point relative to the reference point of the feature map (such as the upper left corner position point of the feature map). Specifically, the corner point position includes the distance from the reference point in the horizontal direction and the distance from the reference point in the vertical direction. The corner point value of the feature point can be the characteristic values ​​of the four corner points of the feature point, which are determined according to the characteristic values ​​of the feature point.

[0129] The so-called key feature region may be a region in the feature map where the features are more obvious. The key feature region may be one or more regions in the feature map.

[0130] Optionally, after obtaining the feature map of the target image, the embodiment may compare the corner point values ​​of each feature point, select the feature points with more obvious features in the feature map, and determine the key feature area in the feature map according to the corner point positions of the feature points with more obvious features. A bilinear interpolation algorithm may also be used to determine the bilinear interpolation of each feature point according to the corner point features of each feature point, and take each feature point with a bilinear interpolation value greater than a preset value as a distinct feature point, and then determine the key feature area in the feature map according to the corner point positions of the distinct feature points.

[0131] S203, encode the key feature area to obtain a feature code stream, and send the feature code stream to the edge end.

[0132] In the embodiment of the present application, the characteristic code stream may be a bit code stream, or a code stream in other forms, etc. Optionally, the form of the characteristic code stream may be determined by the encoding method and the code stream transmission method supported by the communication channel between the terminal and the edge end.

[0133] Optionally, a pre-built visual encoder may be used to encode the key feature area. In one implementation, different visual encoders correspond to different visual tasks. In this case, the required visual encoder may be selected according to the visual task, and then the selected visual encoder may be used to encode the key feature area.

[0134] Furthermore, the encoded feature code stream is sent to the edge through the network, so that the edge can perform the decoding and detection / recognition phases in the visual task processing based on the feature code stream, thereby completing the processing of the entire visual task.

[0135] In the above-mentioned visual task processing method, the key feature area in the feature map of the target image is determined by the corner point features of each feature point in the feature map, and only the determined key feature area is encoded to obtain a feature code stream, and then the feature code stream is sent to the edge end to execute the subsequent process of visual task processing. The above-mentioned scheme, by introducing the corner point features of the feature points in the feature map, accurately extracts the key feature area with obvious features in the feature map, and only encodes the key feature area, thereby achieving the effect of selectively encoding the key feature area in the feature map. Compared with encoding all feature points in the feature map of the target image, the amount of data in the encoded feature code stream is reduced, thereby improving the efficiency of transmitting the feature code stream to the edge end, thereby improving the efficiency of the entire visual task response.

[0136] In one embodiment, in order to improve the efficiency and accuracy of determining the key feature area, this embodiment introduces a method of dividing the feature map into blocks to determine the key feature area, such as Figure 3 As shown, the implementation process of determining the key feature area in the feature map according to the corner point features of each feature point in the feature map in S202 may specifically include the following steps:

[0137] S301, dividing the feature map into at least two feature blocks.

[0138] Optionally, this embodiment can divide the feature map of the target image into two or more feature blocks by non-overlapping uniform division. For example, if the size of the feature map of the target image is n*n, the feature map is divided into m*m feature blocks of the same size by non-overlapping uniform division, and the size of each feature block is Take the target image feature map size of 28*28 as an example. Figure 4As shown, it is evenly divided into 4*4 feature blocks according to non-overlapping, and the size of each feature block is 7*7. After the feature map is divided into 16 feature blocks of the same size, an identification information is assigned to each feature block in turn according to the position of the feature block in the feature map, for example, the identification information of feature block 1 is 1, etc.

[0139] S302, determining a key block from at least two feature blocks according to the corner point features of each feature point in each feature block, and using the key block as a key feature region.

[0140] Optionally, this embodiment can determine, for each feature block, an index value for characterizing the feature importance of the feature block based on the corner point features of each feature point in the feature block, and take the feature blocks whose feature importance index values ​​are greater than a preset feature threshold, or a preset number of feature blocks that are ranked high, as key blocks, and take the areas where the key blocks are located as key feature areas.

[0141] In an optional embodiment, this embodiment may introduce a bilinear interpolation algorithm and use the bilinear average interpolation of each feature block as an index value representing the feature importance of the feature block. Specifically, the following two sub-steps may be included:

[0142] S3021. Using a bilinear interpolation algorithm, determine the bilinear average interpolation of each feature block according to the corner point features of each feature point in each feature block.

[0143] Optionally, this example may adopt a bilinear interpolation algorithm, first determining the bilinear interpolation of each feature point in each feature block based on the corner point features of each feature point in each feature block; and then determining the bilinear average interpolation of each feature block based on the bilinear interpolation of each feature point in each feature block.

[0144] For example, Figure 5 As shown in the figure, the square with 49 boxes on the right is a feature block, and the 49 boxes represent the 49 feature points contained in the feature block. For each feature point (such as feature point 1), its four corner points are L 11 , L 12 , L 22 and L 21 , where the corner point L 11 The corner point position is (ε 1 ,μ 1 ), the corner value is h(L 11 ), corner point L 12 The corner point position is (ε 1 ,μ 2 ), the corner value is h(L 12 ), corner point L22 The corner point position is (ε 2 ,μ 2 ), the corner value is h(L 22 ), corner point L 21 The corner point position is (ε 2 ,μ 1 ), the corner value is h(L 21 ).

[0145] In this embodiment, for each feature point in each feature block, according to the corner point positions and corner point values ​​of the four corner points of the feature point, the corner point L can be calculated by the following formulas (1) and (2): 11 and corner point L 21 Linear interpolation between h(ε,μ 1 ), and the corner point L 12 and corner point L 22 Linear interpolation between h(ε,μ 2 ). Then, according to formula (3), the bilinear interpolation h(ε,μ) of the feature point is calculated.

[0146]

[0147]

[0148]

[0149] Among them, h(ε,μ 1 ) Corner point L representing the feature point 11 and corner point L 21 Linear interpolation between; h(ε,μ 2 ) Corner point L representing the feature point 12 and corner point L 22 Linear interpolation between 1 ,μ 1 ) is the corner point L of the feature point 11 The corner position, (ε 1 ,μ 2 ) is the corner point L of the feature point 12 The corner position of 2 ,μ 2 ) is the corner point L of the feature point 22 The corner position of 2 ,μ 1 ) is the corner point L of the feature point 21 The corner position of h(L 11 ) is the corner point L of the feature point 11 Corner point value; h(L 12 ) is the corner point L of the feature point 12 Corner point value; h(L 21) is the corner point L of the feature point 21 Corner point value; h(L 22 ) is the corner point L of the feature point 22 The corner point value of (ε,μ) is the insertion point of the feature point, and h(ε,μ) is the corner point value of the insertion point, that is, the bilinear interpolation of the feature point.

[0150] After the bilinear interpolation of each feature point in the feature block is calculated based on the above formulas (1) to (3), the bilinear average interpolation of the feature block is calculated based on the bilinear interpolation of each feature point based on formula (4).

[0151]

[0152] Among them, p(j) is the bilinear average interpolation of the jth feature block; i is the i-th feature point in the feature block, h i (ε,μ) is the bilinear interpolation of the i-th feature point; m 2 is the number of feature blocks, n 2 The size of the feature map.

[0153] S3022: Determine a key block from at least two feature blocks according to bilinear average interpolation of each feature block.

[0154] Optionally, after determining the bilinear average interpolation of each feature block, the bilinear average interpolation of each feature block can be compared with a preset bilinear threshold, and the feature block whose bilinear average interpolation is greater than the set bilinear threshold is taken as a key block.

[0155] Alternatively, a preset number of feature blocks with larger bilinear average interpolation values ​​may be selected from at least two feature blocks as key blocks according to the bilinear average interpolation value of each feature block. For example, at least two feature blocks may be sorted in descending order according to the bilinear average interpolation value of each feature block; then, a preset number of feature blocks in the front order may be selected as key blocks.

[0156] like Figure 4 As shown, all feature blocks are sorted from large to small according to the value of the bilinear average interpolation p(j) of each feature block, that is, {p(2), p(7), p(9), p(14), p(16), .....} in sequence, and then according to the sorting results, the top five feature blocks with the highest bilinear average interpolation ranking, namely feature block 2, feature block 7, feature block 9, feature block 14 and feature block 16, are selected as key blocks.

[0157] It can be understood that in this embodiment, by introducing feature blocking and bilinear interpolation determined by the bilinear interpolation algorithm, key blocks with obvious features in the feature map can be accurately extracted, providing data support for selectively encoding key feature areas in the feature map.

[0158] Optionally, the compression ratio used in encoding is an important feature of the encoding process. At present, the compression ratio of the encoder designed based on the deep convolutional neural network structure cannot be adjusted, resulting in inflexible encoding. Based on this, on the basis of the above embodiment, the embodiment of the present application provides an optional method for determining key blocks based on the compression ratio. Figure 6 As shown, the implementation process of determining the key block from at least two feature blocks according to the bilinear average interpolation of each feature block in S3022 may include the following steps:

[0159] S601, determining the number of non-critical blocks according to a preset compression ratio and the number of characteristic blocks.

[0160] Among them, corresponding to the key blocks, the so-called non-key blocks are blocks with unclear characteristic values.

[0161] In the embodiment of the present application, the preset compression ratio can be flexibly configured according to the actual visual task. Optionally, the relationship between the preset compression ratio, the number of feature blocks, and the number of non-critical blocks can be expressed by the following formula (5).

[0162]

[0163] Among them, α represents the preset compression ratio, m 2 (i.e., m*m) represents the number of feature blocks, and q represents the number of non-critical blocks.

[0164] Furthermore, after determining the preset compression ratio and the number of feature blocks, the preset compression ratio and the number of feature blocks can be substituted into formula (5) for solution to obtain the number of non-critical blocks.

[0165] S602: Determine a key block from at least two feature blocks according to the number of non-key blocks and the bilinear average interpolation of each feature block.

[0166] One possible implementation method is to use the difference between the number of feature blocks and the number of non-key blocks as the number of key blocks; and based on the bilinear average interpolation of each feature block, select the feature block with the larger bilinear average interpolation number of key blocks from at least two feature blocks as the key block.

[0167] Another possible implementation method is to sort at least two feature blocks according to the bilinear average interpolation of each feature block; select non-key blocks from at least two feature blocks according to the number of non-key blocks and the sorting result; and use the feature blocks other than the non-key blocks in at least two feature blocks as key blocks. For example, at least two feature blocks can be sorted in descending order according to the bilinear average interpolation of each feature block; according to the sorting result, a preset number of feature blocks in the latter order are used as non-key blocks, where the preset number is the number of non-key blocks; and feature blocks other than the non-key blocks in all feature blocks are used as key blocks.

[0168] It can be understood that the compression ratio is introduced in this embodiment to control the number of non-critical blocks in the feature map, and then control the number of critical blocks in the feature map, thereby increasing the flexibility of the encoding process; further, the compression ratio in this embodiment can be flexibly adjusted according to the actual visual task, thereby broadening the encoding scenario.

[0169] Based on the above embodiments, in one embodiment, Figure 7 As shown, the embodiment of the present application provides an optional method for encoding key blocks. Figure 7 The above S203 encodes the key feature area to obtain the feature code stream, which may include the following steps:

[0170] S701, encoding key blocks in the key feature area to obtain encoded visual blocks.

[0171] In one implementable method, for each key block in the key feature area, block embedding processing (i.e., Patch Embeddings) can be performed on the key block to obtain block embedding information of the key block; then, the block embedding information of the key block can be input into a pre-built visual encoder, which is encoded by the visual encoder to obtain the encoded visual block of the key block.

[0172] In order to obtain a coded visual block with more obvious features, another possible implementation method is to perform linear projection on the key blocks in the key feature area to obtain the block embedding information of the key blocks; through the visual encoder, the block embedding information of the key blocks and the position embedding information of the key blocks in the feature map are encoded to obtain the coded visual block corresponding to the key blocks.

[0173] Among them, the visual encoder in this embodiment is a Transformer-based encoder. The so-called linear projection is the process of converting the key block from a two-dimensional image space to a one-dimensional space. Optionally, the encoding of the block embedding information is essentially the encoding of the feature information contained in the key block. In the application embodiment, the result of encoding the block embedding information is called block encoding data. The so-called position embedding information of the key block in the feature map is used to characterize the position of the key block in the feature map, for example, it can be the identification information of the key block.

[0174] Optionally, for each key block, a linear projection can be performed on the key block to obtain the block embedding information of the key block, and then the block information of the key block is spliced ​​with the position embedding information of the key block in the feature map to obtain the encoding information of the key block. Then, the encoding information of all key blocks is input into the visual encoder together to obtain the encoded visual block corresponding to each key block. Obviously, the encoded visual block corresponding to each key block includes the encoding results of two dimensions: position and block embedding information.

[0175] For example, Figure 8 As shown, for the 16 feature blocks corresponding to the feature map, after selecting key blocks from the 16 feature blocks based on bilinear average interpolation of each feature block, namely feature block 2, feature block 7, feature block 9, feature block 14 and feature block 16, the encoding information of each key block can be input into the visual encoder (i.e., Vision Transformer) to obtain the encoded visual blocks (Encoded VisiblePatches) corresponding to each key block. Figure 8 The white boxes in the figure represent key blocks, the boxes filled with slashes are the encoded visual blocks corresponding to the key blocks (i.e., Encoded Visible Patches), and the black boxes represent non-key blocks in the target image. Since the non-key blocks are not encoded, the non-key blocks here are mask blocks (i.e., Mask Tokens), and their block encoding data can be random values ​​or preset fixed values.

[0176] It can be understood that in this embodiment, by introducing block embedding information and position embedding information for encoding, the visual encoder can extract richer information in the key blocks.

[0177] S702, embedding information according to the positions of the coded visual blocks and non-critical blocks in the feature map to generate a feature code stream.

[0178] Optionally, the encoded visual blocks and the position embedding information of the non-critical blocks in the feature map can be concatenated to obtain a feature code stream.

[0179] Or, for non-critical chunks, Figure 8 For the feature blocks except feature block 2, feature block 7, feature block 9, feature block 14 and feature block 16, a fixed value can be assigned to each non-critical block; then the encoded visual block, as well as the position embedding information of the non-critical block in the feature map and the assigned fixed value can be spliced ​​to obtain the feature code stream.

[0180] It can be understood that in this embodiment, the position of the non-critical blocks in the feature map is introduced into the feature code stream to embed information, so as to facilitate the edge end to quickly reconstruct the feature map of the target image.

[0181] In one embodiment, in order to ensure the accuracy of terminal encoding, the visual encoder in the terminal needs to be updated regularly. Optionally, the terminal receives a trained visual encoder sent by the cloud; wherein the trained visual encoder is trained by the cloud based on a training data set updated by the received decoded visual block; the trained visual encoder is used to replace the local visual encoder. wherein the decoded visual block is obtained by decoding the edge end based on the feature code stream.

[0182] Specifically, each time the edge receives a decoded visual block, it will send it to the cloud. The cloud will update the training data set based on the received decoded visual block. When the updated training data set meets the training conditions, the cloud will use the updated training data set to train the visual encoder and feed the trained visual encoder back to the terminal. After receiving the trained visual encoder sent by the cloud, the terminal will replace the local visual encoder with it, thus completing the update of the local visual encoder of the terminal. This achieves the effect of continuously updating the visual encoder deployed in the Internet of Things.

[0183] Optionally, in the scenario where the visual task processing method of the embodiment of the present application is applied to the cloud, terminal and edge in the visual Internet of Things to realize a certain type of specific visual task through three-party interaction, the terminal will receive the target visual task processing request sent by the edge; and control the image acquisition device to acquire the target image to be encoded corresponding to the target visual task processing request. Among them, the target visual task processing request is generated by the edge based on the original visual task processing request sent by the cloud; the original visual task processing request is generated by the cloud based on the task call request sent by the user.

[0184] Specifically, in this embodiment, when the user has a task call demand, a task call request will be initiated to the cloud through the user terminal. The cloud will generate an original visual task processing request based on the task call request and send it to the edge terminal. The edge terminal can directly send the original visual task processing request as a target visual task processing request to the terminal, or it can generate a new target visual task processing request after processing the original visual task processing request and send it to the terminal. For example, the original visual task processing request can be parsed to analyze the processing tasks (such as image acquisition and visual encoding tasks) that need to be performed by the terminal, and then the target visual task processing request can be regenerated based on the processing tasks and sent to the terminal. After the terminal receives the target visual task processing request, it will control the image acquisition device in the visual Internet of Things to collect the target image to be encoded corresponding to the target visual task processing request. For example, the terminal will send an image acquisition instruction to the image acquisition device, and the image acquisition device will respond to the image acquisition instruction, perform the acquisition operation of the target image, and feed back the acquired target image to the terminal.

[0185] In one embodiment, Fig. 9 As shown, a visual task processing method is provided, which is applied to Figure 1 Take the edge end in as an example, which is mainly used to perform the feature decoding part in the visual task. It includes the following steps:

[0186] S901, obtaining a feature code stream sent by a terminal.

[0187] The so-called feature code stream is obtained by encoding the key feature area by the terminal, and the key feature area is determined by the terminal according to the corner features of each feature point in the feature map of the target image to be encoded. It should be noted that the specific generation process of the feature code stream has been introduced in the above embodiment and will not be repeated here.

[0188] Optionally, since the visual processing task of the embodiment of the present application is a visual processing task based on the visual Internet of Things, the terminal in the above embodiment encodes the feature map of the target image to be encoded, and after obtaining the feature code stream, the feature code stream is transmitted to the edge end of the visual Internet of Things for performing the decoding task. Correspondingly, the edge end will receive the feature code stream sent by the terminal.

[0189] S902, decoding the feature code stream to obtain a decoded visual block.

[0190] The so-called decoded visual block is the decoding result of the feature code stream, which can be used to reconstruct the feature map of the target image to be encoded.

[0191] Optionally, this embodiment can be a coded visual block obtained by parsing the key feature area (i.e., key block) of the feature map of the target image by the terminal from the feature code stream, and then inputting the parsed coded visual block into a visual decoder matching the feature encoder on the terminal side, and the visual decoder decodes the input coded visual block to obtain a decoded visual block. Specifically, the above embodiment has introduced that the terminal encodes the block embedding information of the key block and the position embedding information of the key block in the target image to generate a coded visual block, so when the visual decoder of this embodiment performs a decoding operation on the coded visual block, on the one hand, it parses the decoded visual block corresponding to each coded visual block, and on the other hand, based on the position embedding information corresponding to each coded visual block, parses the position of the non-key block corresponding to the non-key feature area in the feature map, and decodes the decoded visual block corresponding to the non-key block according to the position of each non-key block and the coded block data of the coded visual block near the key block.

[0192] In the visual task processing method of the embodiment of the present application, the feature code stream received by the edge end is obtained by encoding the key feature area of ​​the feature map in the target image to be encoded by the terminal. Since the feature coding stream only contains the visual coding features of the key feature areas in the feature map, this embodiment does not need to transmit the visual coding features of all feature areas in the feature map, which reduces the transmission amount and receiving efficiency of the feature code stream. In addition, when feature decoding is performed based on the feature code stream, the decoded visual block of the entire feature map can be reconstructed based only on the visual coding features of the key feature areas, which realizes the completion of the image feature decoding part in the visual task on the basis of transmitting a small amount of feature code stream, and provides a new solution for the interaction between the terminal and the edge end based on the visual Internet of Things to realize the image feature encoding and decoding operations in the visual task.

[0193] Optionally, when the visual task processing method of the present application is applied to a scenario in which the cloud, terminal and edge in the visual Internet of Things interact to complete a certain type of specific visual task (such as target recognition or target detection task), the edge will receive the original visual task processing request sent by the cloud, and based on the original visual task processing request, send a target visual task processing request to the terminal; wherein the original visual task processing request is generated by the cloud based on the task call request sent by the user; the target visual task processing request is used to request the terminal to feedback the feature code stream. Correspondingly, after the edge decodes the feature decoding stream to obtain the decoded visual block, it will also perform the visual task according to the decoded visual block, obtain the task processing result, and feedback the task processing result to the user.

[0194] Specifically, in this embodiment, when a user has a task call requirement, a task call request will be initiated to the cloud through the user end. The cloud will generate an original visual task processing request based on the task call request and send it to the edge end. The edge end can directly send the original visual task processing request as a target visual task processing request to the terminal, and can also process the original visual task processing request and generate a new target visual task processing request and send it to the terminal. For example, the original visual task processing request can be parsed to analyze the processing tasks that need to be performed by the terminal (such as image acquisition and visual encoding tasks), and then the target visual task processing request can be regenerated based on the processing task and sent to the terminal. After receiving the target visual task processing request, the terminal will control the image acquisition device to acquire the target image to be encoded corresponding to the target visual task processing request, extract the feature map of the target image, and determine the key feature area in the feature map according to the corner features of each feature point in the feature map; encode the key feature area to obtain a feature code stream, and send the feature code stream as the response result of the target visual task to the edge end. Since the edge end has only completed the decoding part of the visual task after decoding the feature decoding stream to obtain the decoded visual block, it is also necessary to directly send the obtained decoded visual block to the downstream machine vision task for the visual Internet of Things, that is, continue to perform the downstream visual task based on the decoded visual block to obtain the task processing result. For example, the visual algorithm corresponding to the visual task (such as a target recognition algorithm or a target detection algorithm) can be called to perform visual task processing (such as target recognition processing or target detection processing) based on the feature map reconstructed by the decoded visual block to obtain the task processing result of the visual task. Since the visual task is essentially generated based on the task call request of the user end, the edge end can directly or indirectly feedback the task processing result to the user end through the cloud after obtaining the task processing result. In order to provide visual task processing services for the user end. Optionally, the edge end can not only feed back the task processing results to the user end, but also perform statistical analysis of the visual task and display the analysis results, etc. There is no limitation on this.

[0195] In one embodiment, in order to ensure that the visual decoder can decode the decoded visual block corresponding to the feature map more accurately, when generating the feature code stream, the terminal of the above embodiment can choose to generate the feature code stream based on the encoded visual block obtained by encoding the key block in the key feature area and the position embedding information of the non-key block in the feature map. Fig.10 and Fig.11 As shown, the edge end executes S902 to decode the feature code stream, and the specific steps of obtaining the decoded visual block may include:

[0196] S1001, obtaining the position embedding information of the coded visual block and the non-critical block in the feature map from the feature code stream.

[0197] Optionally, since the feature code stream of this embodiment is generated by the terminal based on the coded visual blocks obtained by encoding the key blocks in the key feature area and the position embedding information of the non-key blocks in the feature map, the edge side parses the feature code stream to obtain the position embedding information of the coded visual blocks and the non-key blocks in the feature map.

[0198] S1002, decoding the position embedding information of the encoded visual block and the non-critical block in the feature map through a visual decoder to obtain a decoded visual block.

[0199] Specifically, this embodiment can input the position embedding information of the encoded visual block and the non-critical block in the feature map into the visual decoder together. The visual decoder can not only decode the encoded visual block to obtain the decoded visual block corresponding to the key block in the feature map, but also decode the decoded visual block corresponding to the non-critical block in the feature map according to the position embedding information of the non-critical block in the feature map and the encoded visual block.

[0200] For example, Fig.11 As shown, the Transformer feature decoder is used as the visual decoder in this step. Fig.11 The square filled with diagonal lines in the figure represents the encoded visual patches corresponding to the key patches, and the black square marks represent the non-key patches in the feature map of the target image. Since the non-key patches are not encoded, the non-key patches here are mask patches, and their encoded patch data can be random values ​​or preset fixed values. This embodiment can be Fig.11 The coded visual blocks and mask blocks in the feature map are embedded with their position information in the feature map and then input into the first normalization network of the Transformer feature decoder. Based on the first normalization network, each coded block (i.e., each coded visual block and mask block) is first normalized to obtain the first normalization result of each coded block. Then, for the first normalization result of each coded block, the three matrices Q corresponding to each coded block are determined according to the following formula (6): t , K t 、V t .

[0201]

[0202] Where t is the first normalization processing result corresponding to each coding block, [W t Q ,W t K ,W t V] is the attention weight matrix of each head in the randomly initialized multi-head self attention mechanism in the Transformer feature decoder, Q t , K t and V t These are the three matrices corresponding to each coding block.

[0203] The three matrices Q corresponding to each coding block t , K t and V t Input into the randomly initialized Multi-head Self-Attention network, which determines the multi-head value corresponding to each encoding block based on the following formula (7)_ and formula (8).

[0204]

[0205]

[0206] Among them, s t The encoding block t corresponds to each attention head, is the dimension of the matrix Kt, δ() is the function for calculating attention, and τ() is the Softmax logistic regression function. is the connection function (i.e. Concatenate function), W t e is the parameter matrix, and r is the multi-head value corresponding to the encoding block t.

[0207] The multi-head value corresponding to each coding block, the block coding data and position embedding information of the coding block are fused, and the fusion result is input into the second normalization network, and the second normalization result output by the second normalization network is input into the multi-layer perception (MLP) network of the Transformer feature decoder to obtain the perception result of each coding block, and finally the perception result of each coding block is input into the normalization (Softmax) network of the Transformer feature decoder to obtain the decoding result of each coding block, that is, the decoding visual block corresponding to each coding block. The reconstructed feature map can be obtained through each decoding visual block.

[0208] In this implementation, the feature code stream received by the edge end includes not only the encoded visual block, but also the position embedding information of the non-critical block in the feature map. Since the encoded visual block also includes the position embedding information of the non-critical block in the feature map, the visual decoder of this embodiment uses the original position embedding information of the critical block and the non-critical block in the feature map during the feature decoding process. This not only ensures the accuracy of decoding the critical block, but also accurately locates the other blocks around each non-critical block, thereby determining the decoded visual block of each non-critical block based on the block encoding data of other surrounding blocks, thereby improving the accuracy of each decoded visual block finally decoded.

[0209] In one embodiment, in order to ensure the accuracy of the edge decoding result, the visual decoder at the edge needs to be regularly updated and trained. Specifically, the embodiment of the present application can introduce the cloud, and complete the update training of the visual decoder at the edge through the cooperation between the cloud and the edge. Specifically, after the feature code stream is decoded in S902 and the decoded visual block is obtained, the edge can send the decoded visual block to the cloud, and the decoded visual block is used for the cloud to update the training data set. Correspondingly, the subsequent edge may receive the trained visual decoder sent by the cloud, and use the trained visual decoder to replace the local visual decoder, wherein the trained visual decoder is trained based on the updated training data set. Specifically, each time the edge obtains the decoded visual block, it will be sent to the cloud, and the cloud will update the training data set based on the received decoded visual block. When the updated training data set meets the training conditions, the cloud will use the updated training data set to train the visual decoder, and feed back the trained visual decoder to the edge. After receiving the trained visual decoder sent by the cloud, the edge will replace the local visual decoder with it to complete the update of the local visual decoder at the edge. This enables the effect of continuous updating of the visual decoder deployed in the Internet of Things.

[0210] Optionally, if the edge end performs a visual task after obtaining the decoded visual block and obtains a task processing result, in order to ensure the accuracy of the cloud's training of the visual decoder, the edge end can send the task processing result at the same time as sending the decoded visual block to the cloud, so that the cloud can determine whether to use the decoded visual block to update the training data set based on the task processing result, thereby ensuring the accuracy of the updated training data set, and thereby improving the accuracy of the visual decoder trained based on the training data set.

[0211] It should be noted that the process of how the cloud updates the training data set based on the decoded visual blocks (or decoded visual blocks and task processing results) sent by the edge, and how to continue training the visual decoder based on the updated training data set will be described in detail in subsequent embodiments and will not be repeated here.

[0212] In one embodiment, Fig.12 As shown, a visual task processing method is provided, which is applied to Figure 1 The cloud in the example is used to connect the user end and the edge end to assist in the visual task processing based on the visual Internet of Things. It includes the following steps:

[0213] S1201, receiving a task calling request sent by a user terminal.

[0214] Among them, the task call request can be a request initiated by the user end to request the visual Internet of Things to call the machine vision algorithm to perform the machine vision task when the user has a machine vision task processing demand.

[0215] Optionally, in this embodiment, when a user has a need to process a machine vision task, the user can select the type of machine vision task to be executed on the user terminal, and the user terminal can generate a task call request according to the type of machine vision task selected by the user, and send it to the cloud of the visual Internet of Things that can execute the machine vision task. Correspondingly, the cloud of the visual Internet of Things can receive the task call request sent by the user terminal.

[0216] S1202: Based on the task calling request, an original visual task processing request is sent to the edge end.

[0217] Among them, the original visual task processing request is used to instruct the edge end to send a target visual task processing request to the terminal; the target visual task processing request is used to instruct the terminal to encode the key feature area in the feature map according to the corner point features of each feature point in the feature map of the target image to be encoded to obtain a feature code stream, and the edge end determines the task processing result according to the feature code stream. It should be noted that how the edge end initiates a target visual task processing request to the terminal according to the original visual task processing request, how the terminal responds to the target visual task request and sends the visually encoded feature code stream to the edge end, and how the edge end determines the task processing result according to the feature code stream has been introduced in the above embodiments and will not be repeated here.

[0218] Optionally, the cloud sends the original task processing request to the edge based on the task call request by parsing the task call request sent by the user end, determining the type of machine learning task that the user end needs to call, and then determining the edge end that executes the machine learning task in the visual Internet of Things based on the parsing result, and initiating the original task processing request to the corresponding edge end in accordance with the communication standard of the visual Internet of Things.

[0219] In the above-mentioned visual task processing method, the cloud triggers the start of the visual task processing process by interacting with the edge; in the visual task processing process, the terminal accurately extracts the key feature area with obvious features in the feature map based on the corner features of the feature points in the feature map, and only encodes the key feature area, thereby achieving the effect of selectively encoding the key feature area in the feature map. Compared with encoding all feature points in the feature map of the target image, the amount of data of the encoded feature code stream is reduced, thereby improving the efficiency of transmitting the feature code stream to the edge, thereby improving the efficiency of the entire visual task response.

[0220] Optionally, in order to ensure the security of visual task execution, after receiving the task call request sent by the user end, the cloud can also obtain the user information associated with the user end according to the task call request; perform permission verification on the user information; and send the original visual task processing request to the edge end if the permission verification passes. Specifically, if the task call request contains user information, the user information contained in the task call request can be used as the user information associated with the user end; if the task call request does not contain user information, the information of the user end that sent the task call request can be used as the user information associated with the user end. When the user information is verified for permission, the user information associated with the user end obtained at this time can be compared with the authorized user information pre-stored in the cloud. If the user information associated with the user end belongs to the authorized user information, it is determined that the user information permission verification is passed, and the original visual task processing request can be sent to the edge end at this time. Otherwise, it is considered that the user information permission verification fails, and the subsequent operation of initiating the original visual task processing request to the edge end is not performed. In this embodiment, the cloud verifies the user information authority of the task call request initiated by the user end and then initiates the original visual task processing request to the cloud of the visual Internet of Things, thereby preventing unauthorized users from arbitrarily triggering the visual Internet of Things to perform visual tasks, thereby improving the security of visual task execution.

[0221] In one embodiment, in order to ensure the accuracy of the edge decoding results, the visual decoder and / or the visual encoder need to be regularly updated and trained. Specifically, the embodiment of the present application can introduce cooperation between the cloud and the edge to complete the update training of the visual decoder and / or the visual encoder. Fig.13As shown, after the method sends the original visual task processing request to the edge end in S1202, it also includes the following steps:

[0222] S1301, receiving a decoded visual block sent by an edge end.

[0223] Optionally, after the cloud sends the original visual task processing request to the edge, the edge will interact with the terminal to execute the original visual task processing request (the specific implementation method has been introduced in the above embodiment and will not be repeated here). During this process, the edge will generate a decoded visual block and a task processing result corresponding to the original visual task processing request. In this embodiment, the edge can only feedback the decoded visual block obtained when executing the original visual task processing request to the cloud after each execution of the original visual task processing request. It can also feedback the decoded visual block to the cloud while feeding back the task processing result. There is no limitation on this. Accordingly, the cloud will receive the decoded visual block or the decoded visual block and the task processing result fed back by the edge each time the original visual task processing request is sent to the edge.

[0224] S1302, using the decoded visual block to update the training data set.

[0225] Specifically, after receiving the decoded visual block fed back by the edge, the cloud can first reconstruct the decoded feature map based on the decoded visual block, and then directly add the reconstructed feature map to the training data set to achieve an update of the training data set.

[0226] Optionally, if the cloud not only obtains the decoded visual block but also obtains the task processing result in step S1301, then this step can also first determine whether the decoded visual block received this time meets the training set update condition based on the task processing result; if it does, the decoded visual block is used to update the training data set. Among them, the training set update condition can be updated when a positive sample is encountered.

[0227] Specifically, according to the task processing result, it is determined whether the feature map reconstructed by the decoded visual block received this time belongs to a positive sample. If so, it is determined that the training set update condition is met, and the decoded visual block can be continued to be used to update the training data set. For example, if the visual task of this embodiment is a defect recognition task, then if the task processing result is that there is a defect, it is determined that the received decoded visual block meets the training set update condition, and the operation of using the decoded visual block to update the training data set can be performed; if the task processing result is that there is no defect, it is determined that the received decoded visual block does not meet the training set update condition, that is, the decoded visual block cannot be used to update the training data set.

[0228] S1303: When the updated training data set meets the training condition, the visual encoder and / or the visual decoder are trained using the updated training data set.

[0229] It should be noted that in this embodiment, after the cloud sends the original visual task processing request to the edge each time, it will update the training data set based on the decoded visual block received from the edge feedback (or update the training data set when it is determined that the training set update conditions are met based on the task processing results). After each update of the training data set, it will trigger a judgment on whether the updated training data set meets the training conditions. For example, it can be judged whether the total amount of data in the updated training data set reaches a preset number. If so, it is judged that the training conditions are met. It can also be judged whether the time interval between the training data set and the last time the training conditions were met reaches a preset time length. If so, it is determined that the training conditions are met.

[0230] If the updated training data set meets the training conditions, the updated training data set will be used as a training sample to perform supervised or unsupervised training on the visual encoder and / or visual decoder trained last time on the cloud. If the updated training data set does not meet the training conditions, this step and subsequent operations will not be performed. It should be noted that if the cloud is training the visual encoder and / or visual decoder for the first time, the training objects at this time can be the visual decoder and visual encoder deployed at the edge and terminal in the visual Internet of Things.

[0231] It should also be noted that in this embodiment, after the cloud uses the updated training data set to train the visual encoder and / or visual decoder, it can clear the training data set so that subsequent iterative updates can obtain a new training data set to train the visual encoder and / or visual decoder.

[0232] S1304: Send the trained visual decoder to the edge end, and / or send the trained visual encoder to the terminal.

[0233] Optionally, in this embodiment, the cloud can send the visual decoder of this training to the edge of the visual Internet of Things, and send the visual encoder of this training to the terminal of the visual Internet of Things after each training. It can also be that after each training, it is determined whether the accuracy of the visual decoder and visual encoder before and after training has improved. If the accuracy has improved, it will be sent to the corresponding edge or terminal. For example, if the accuracy of the visual decoder after training is higher than that of the visual decoder before training, the visual decoder is sent to the edge. If the accuracy of the visual encoder after training is almost unchanged from the accuracy of the visual encoder before training, or the accuracy is reduced, the visual encoder after this training will not be sent to the terminal.

[0234] In this embodiment, the cloud obtains the decoded visual blocks obtained by the edge end when connecting to the user end and the edge end to assist in the processing of visual tasks based on the visual Internet of Things, and updates the training data set based on the decoded visual blocks. When the training data set meets the training conditions, the visual encoder and / or visual decoder are trained, and the trained visual encoder and / or visual decoder are fed back to the corresponding terminal and / or edge end. In the process of the visual Internet of Things performing visual tasks, the visual encoder deployed in the terminal of the visual Internet of Things and the visual decoder deployed at the edge end are continuously updated, which provides a guarantee for the accurate execution of visual tasks by the visual Internet of Things.

[0235] In one embodiment, the present application provides a bilinear interpolation-based Transformer encoding and decoding system, which is essentially an Internet of Things-based encoding and decoding system. Fig.13 As shown, the system includes a cloud, an edge, and a terminal.

[0236] Among them, the cloud is at the application layer of the system and can be used to connect to the user end. For example, the cloud can receive the user's task call request through the user end; further, the cloud can also be used to connect to the edge end to trigger the edge end to start the processing flow of the visual task. For example, the cloud can send the original visual task processing request to the edge end, so that the edge end can trigger the terminal to execute the encoding process in the visual task processing process based on the original visual task processing request. In addition, the cloud can also regularly update the Transformer-based visual decoder integrated in the edge end. Furthermore, the cloud can also connect to the terminal, for example, the cloud can regularly update the Transformer-based visual encoder integrated in the terminal.

[0237] The edge is located in the service and application support layer of the system, and is mainly used to perform the decoding process and specific machine vision tasks (such as target detection / recognition) in the visual task processing process. Optionally, the edge can be used to connect to the cloud, for example, to obtain the original visual task processing request sent by the cloud, to send to the cloud the decoded visual block obtained by decoding the feature code stream sent by the terminal, and to feedback the task processing results of the machine vision task to the cloud.

[0238] Furthermore, the edge end can also be connected to a terminal located at the device layer of the system. For example, a target visual task processing request can be sent to the terminal to trigger the terminal to execute the encoding process in the entire visual task processing process.

[0239] exist Fig.14 Based on the system shown in FIG. 1 , the embodiment of the present application provides a process for visual task processing to be realized by mutual interaction between the user end, the cloud end, the edge end and the terminal. Fig.15As shown, the method includes the following steps.

[0240] S1501, the user terminal sends a task call request to the cloud.

[0241] S1502, the cloud receives a task invocation request sent by the user.

[0242] S1503, the cloud obtains the user information associated with the user end according to the task call request, and performs permission verification on the user information.

[0243] S1504: When the authority verification is passed, the cloud sends the original visual task processing request to the edge.

[0244] S1505, the edge receives the original visual task processing request sent by the cloud.

[0245] S1506: The edge sends a target visual task processing request to the terminal based on the original visual task processing request.

[0246] S1507, the terminal receives the target visual task processing request sent by the edge end.

[0247] S1508, the terminal controls the image acquisition device to acquire the target image to be encoded corresponding to the target visual task processing request, and performs feature extraction on the target image to obtain a feature map.

[0248] S1509: The terminal divides the feature map into at least two feature blocks, and uses a bilinear interpolation algorithm to determine a bilinear average interpolation of each feature block according to the corner point features of each feature point in each feature block.

[0249] S1510, the terminal determines the number of non-critical blocks according to a preset compression ratio and the number of characteristic blocks, and determines a critical block from at least two characteristic blocks according to the number of non-critical blocks and bilinear average interpolation of each characteristic block.

[0250] S1511, the terminal encodes the key blocks to obtain the encoded visual blocks, embeds information according to the positions of the encoded visual blocks and the non-key blocks in the feature map, and generates a feature code stream.

[0251] S1512: The terminal sends the feature code stream to the edge end.

[0252] S1513: The edge end obtains the feature code stream sent by the terminal.

[0253] S1514, the edge end obtains the position embedding information of the coded visual block and the non-critical block in the feature map from the feature code stream.

[0254] S1515, the edge end decodes the position embedding information of the encoded visual block and the non-critical block in the feature map through the visual decoder to obtain the decoded visual block, performs the visual task according to the decoded visual block, obtains the task processing result, and feeds back the task processing result to the user end.

[0255] Optionally, the edge terminal can not only feed back the task processing result to the user terminal, but also perform statistical analysis of the visual task and display the analysis result, etc. There is no limitation on this.

[0256] S1516, the edge sends the decoded visual block to the cloud.

[0257] S1517, the cloud receives the decoded visual block sent by the edge.

[0258] S1518, the cloud uses the decoding vision block to update the training data set, and when the updated training data set meets the training conditions, the updated training data set is used to train the visual encoder and the visual decoder.

[0259] S1519, the cloud sends the trained visual decoder to the edge, and sends the trained visual encoder to the terminal.

[0260] S1520, the terminal receives the trained visual encoder sent by the cloud, and uses the trained visual encoder to replace the local visual encoder.

[0261] S1521, the edge receives the trained visual decoder sent by the cloud, and uses the trained visual decoder to replace the local visual decoder.

[0262] It should be understood that although Figure 1-15 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1-15 At least part of the steps may include multiple steps or multiple stages. These steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed in turn or alternately with other steps or at least part of the steps or stages in other steps.

[0263] In one embodiment, Fig.16 As shown, a visual task processing device 1 is provided, which is configured in a terminal and includes: a feature extraction module 10, a region determination module 11 and a coding transmission module 12, wherein:

[0264] A feature extraction module 10 is used to extract features from a target image to be encoded to obtain a feature map;

[0265] A region determination module 11 is used to determine a key feature region in a feature map according to corner point features of each feature point in the feature map;

[0266] The encoding and transmission module 12 is used to encode the key feature area to obtain a feature code stream, and send the feature code stream to the edge end.

[0267] In one embodiment, the region determination module 11 includes:

[0268] A feature block unit, used for dividing the feature map into at least two feature blocks;

[0269] The key area determination unit is used to determine the key block from at least two feature blocks according to the corner point features of each feature point in each feature block, and use the key block as the key feature area.

[0270] In one embodiment, the key area determination unit includes:

[0271] The interpolation subunit is used to determine the bilinear average interpolation of each feature block according to the corner point features of each feature point in each feature block by using a bilinear interpolation algorithm;

[0272] The region screening subunit is used to determine a key block from at least two feature blocks according to bilinear average interpolation of each feature block.

[0273] In one of the embodiments, the above-mentioned interpolation subunit is specifically used to: use a bilinear interpolation algorithm to determine the bilinear interpolation of each feature point in each feature block according to the corner point features of each feature point in each feature block; and determine the bilinear average interpolation of each feature block according to the bilinear interpolation of each feature point in each feature block.

[0274] In one embodiment, the corner point features include: corner point position and corner point value.

[0275] In one of the embodiments, the above-mentioned area screening subunit is specifically used to: determine the number of non-critical blocks based on a preset compression ratio and the number of feature blocks; determine the critical blocks from at least two feature blocks based on the number of non-critical blocks and bilinear average interpolation of each feature block.

[0276] In one of the embodiments, when the region screening subunit performs bilinear average interpolation based on the number of non-critical blocks and each feature block to determine the critical block from at least two feature blocks, it is also specifically used to: sort the at least two feature blocks based on the bilinear average interpolation of each feature block; select the non-critical block from the at least two feature blocks based on the number of non-critical blocks and the sorting result; and use the feature blocks other than the non-critical blocks in the at least two feature blocks as the critical blocks.

[0277] In one embodiment, the encoding transmission module 12 includes:

[0278] An encoding unit, used for encoding key blocks in the key feature area to obtain encoded visual blocks;

[0279] The code stream generation unit is used to generate a feature code stream according to the position embedding information of the coded visual block and the non-critical block in the feature map.

[0280] In one of the embodiments, the above-mentioned encoding unit is specifically used to: perform linear projection on the key block in the key feature area to obtain block embedding information of the key block; encode the block embedding information of the key block and the position embedding information of the key block in the feature map through a visual encoder to obtain the encoded visual block corresponding to the key block.

[0281] In one embodiment, the visual task processing device further includes:

[0282] The encoder receiving module is used to receive the trained visual encoder sent by the cloud; wherein the trained visual encoder is obtained by training the training data set updated by the cloud based on the received decoded visual block; the decoded visual block is obtained by decoding the edge end based on the feature code stream;

[0283] The encoder replacement module is used to replace the local visual encoder with the trained visual encoder.

[0284] In one embodiment, the visual task processing device further includes:

[0285] A target receiving module is used to receive a target visual task processing request sent by the edge end;

[0286] The image acquisition module is used to control the image acquisition device to acquire the target image to be encoded corresponding to the target visual task processing request.

[0287] Among them, the target visual task processing request is generated by the edge based on the original visual task processing request sent by the cloud; the original visual task processing request is generated by the cloud based on the task call request sent by the user.

[0288] In one embodiment, Fig.17 As shown, a visual task processing device 2 is provided, which is configured at the edge end and includes: a code stream acquisition module 20 and a decoding module 21, wherein:

[0289] The code stream acquisition module 20 is used to acquire the characteristic code stream sent by the terminal; wherein the characteristic code stream is obtained by encoding the key characteristic area by the terminal, and the key characteristic area is determined by the terminal according to the corner point features of each characteristic point in the characteristic map of the target image to be encoded;

[0290] The decoding module 21 is used to decode the feature code stream to obtain a decoded visual block.

[0291] In one of the embodiments, the feature code stream is generated by the terminal according to the encoded visual block obtained by encoding the key block in the key feature area, and the position embedding information of the non-key block in the feature map;

[0292] Correspondingly, the above decoding module is specifically used to: obtain the position embedding information of the encoded visual block and the non-critical block in the feature map from the feature code stream; through the visual decoder, decode the position embedding information of the encoded visual block and the non-critical block in the feature map to obtain the decoded visual block.

[0293] In one embodiment, the visual task processing device further includes:

[0294] A decoding block sending module is used to send a decoded visual block to the cloud; wherein the decoded visual block is used for the cloud to update the training data set;

[0295] A decoder receiving module, used to receive a trained visual decoder sent by the cloud; wherein the trained visual decoder is trained based on an updated training data set;

[0296] The decoder replacement module is used to replace the local visual decoder with the trained visual decoder.

[0297] In one embodiment, the visual task processing device further includes:

[0298] An original receiving module, used to receive an original visual task processing request sent by the cloud; wherein the original visual task processing request is generated by the cloud based on the task call request sent by the user end;

[0299] A target sending module is used to send a target visual task processing request to the terminal based on the original visual task processing request; wherein the target visual task processing request is used to request the terminal to feedback a feature code stream;

[0300] The task execution module is used to execute the visual task according to the decoded visual block and obtain the task processing result;

[0301] The task result feedback module is used to feedback the task processing results to the user end.

[0302] In one embodiment, Fig.18 As shown, a visual task processing device 3 is provided, which is configured in the cloud and includes: a request receiving module 30 and a request sending module 31, wherein:

[0303] The request receiving module 30 is used to receive the task calling request sent by the user end;

[0304] The request sending module 31 is used to send an original visual task processing request to the edge end based on the task call request; the original visual task processing request is used to instruct the edge end to send a target visual task processing request to the terminal; the target visual task processing request is used to instruct the terminal to encode the key feature area in the feature map according to the corner point features of each feature point in the feature map of the target image to be encoded to obtain a feature code stream, and the edge end determines the task processing result according to the feature code stream.

[0305] In one of the embodiments, the request sending module 31 is specifically used to: obtain user information associated with the user end according to the task call request; perform authority verification on the user information; and send the original visual task processing request to the edge end when the authority verification passes.

[0306] In one of the embodiments, the above-mentioned visual task processing device also includes a training module, which is specifically used to: receive a decoded visual block sent by an edge end; use the decoded visual block to update a training data set; when the updated training data set meets the training conditions, use the updated training data set to train the visual encoder and / or visual decoder; send the trained visual decoder to the edge end, and / or send the trained visual encoder to the terminal.

[0307] For the specific definition of the visual task processing device, please refer to the definition of the visual task processing device method above, which will not be repeated here. Each module in the above-mentioned visual task processing device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0308] Fig.19A schematic diagram of the structure of an edge / terminal device provided for an embodiment of the present application. The edge / terminal device 4 may include a receiver 41, a memory 42, a processor 43, at least one communication bus 44 and a transmitter 45. The communication bus 44 is used to realize the communication connection between components. The memory 42 may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory. Various programs may be stored in the memory 42 to complete various processing functions and implement the method steps of this embodiment. In this embodiment, the transmitter 45 may be a radio frequency processing module or a baseband processing module in an access network device, and the receiver 41 may also be a radio frequency processing module or a baseband processing module in an access network device. The transmitter 45 and the receiver 41 can be integrated together to realize a transceiver, and the transmitter 45 and the receiver 41 can be coupled to the processor 43, which can realize the receiving or sending action under the instruction or control of the processor 43.

[0309] In this embodiment, if Fig.19 When it is a terminal device, the processor 43 is used to extract features of the target image to be encoded to obtain a feature map; determine the key feature area in the feature map according to the corner point features of each feature point in the feature map; encode the key feature area to obtain a feature code stream;

[0310] The transmitter 45 is used to send the feature code stream to the edge end.

[0311] In this embodiment, if Fig.19 When it is an edge device, the receiver 41 is used to obtain a feature code stream sent by the terminal; wherein the feature code stream is obtained by encoding the key feature area by the terminal, and the key feature area is determined by the terminal according to the corner point features of each feature point in the feature map of the target image to be encoded;

[0312] The processor 43 is used to decode the feature code stream to obtain a decoded visual block.

[0313] In one embodiment, a cloud device is provided. Fig. 20 . Fig. 20 It is a structural diagram of a cloud device provided in an embodiment of the present invention. Fig. 20 The cloud device 5 shown includes: at least one processor 51, a memory 52, at least one network interface 54 and a user interface 53. The various components in the cloud device 5 are coupled together via a bus system 55. It is understood that the bus system 55 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 55 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, Fig. 20Various buses are labeled as bus system 55. In addition, in the embodiment of the present invention, a transceiver 56 is also included. The transceiver can be a plurality of components, that is, including a transmitter and a receiver, and is a unit for communicating with various other devices on a transmission medium.

[0314] The user interface 53 may include a display, a keyboard, or a pointing device (eg, a mouse, a trackball, a touch pad, or a touch screen).

[0315] It can be understood that the memory 52 in the embodiment of the present invention can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (DRRAM). The memory 52 of the system and method described in the embodiments of the present invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0316] In some implementations, the memory 52 stores the following elements, executable modules or data structures, or a subset thereof, or an extended set thereof: an operating system 521 and application programs 522 .

[0317] The operating system 521 includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic services and process hardware-based tasks. The application 522 includes various application programs, such as a media player (MediaPlayer), a browser (Browser), etc., which are used to implement various application services. The program for implementing the method of the embodiment of the present invention can be included in the application 522.

[0318] In an embodiment of the present invention, by calling a program or instruction stored in the memory 52, specifically, the program or instruction stored in the application 522, the receiver is used to receive a task call request sent by the user end; the transmitter is used to send an original visual task processing request to the edge end based on the task call request; wherein the original visual task processing request is used to instruct the edge end to send a target visual task processing request to the terminal; the target visual task processing request is used to instruct the terminal to encode the key feature area in the feature map according to the corner point features of each feature point in the feature map of the target image to be encoded to obtain a feature code stream, and the edge end determines the task processing result according to the feature code stream.

[0319] It is to be understood that the embodiments described in the embodiments of the present invention can be implemented by hardware, software, firmware, middleware, microcode or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application specific integrated circuits (Application Specific Integrated Circuits, ASIC), digital signal processors (Digital Signal Processing, DSP), digital signal processing devices (DSP Device, DSPD), programmable logic devices (Programmable Logic Device, PLD), field programmable gate arrays (Field-Programmable Gate Array, FPGA), general processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions of the present application or a combination thereof.

[0320] For software implementation, the technology of the embodiments of the present invention can be implemented by modules (such as procedures, functions, etc.) that perform the functions of the embodiments of the present invention. The software code can be stored in a memory and executed by the processor 51. The memory can be implemented in the processor 51 or outside the processor 51.

[0321] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor of a terminal, the following steps are implemented:

[0322] Perform feature extraction on the target image to be encoded to obtain a feature map;

[0323] Determine a key feature area in the feature map according to the corner point features of each feature point in the feature map;

[0324] The key feature area is encoded to obtain a feature code stream, and the feature code stream is sent to the edge end.

[0325] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor at an edge end, the following steps are implemented:

[0326] Acquire a feature code stream sent by the terminal; wherein the feature code stream is obtained by encoding a key feature area by the terminal, and the key feature area is determined by the terminal according to corner point features of each feature point in a feature map of a target image to be encoded;

[0327] The feature code stream is decoded to obtain a decoded visual block.

[0328] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor in the cloud, the following steps are implemented:

[0329] Receive the task call request sent by the user;

[0330] Based on the task call request, an original visual task processing request is sent to the edge end; wherein the original visual task processing request is used to instruct the edge end to send a target visual task processing request to the terminal; the target visual task processing request is used to instruct the terminal to encode the key feature area in the feature map of the target image to be encoded according to the corner point features of each feature point in the feature map to obtain a feature code stream, and the edge end determines the task processing result according to the feature code stream.

[0331] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0332] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0333] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A visual task processing method, characterized in that: For a terminal, the method comprises: Perform feature extraction on the target image to be encoded to obtain a feature map; Determine a key feature area in the feature map according to the corner point features of each feature point in the feature map; The key feature area is encoded to obtain a feature code stream, and the feature code stream is sent to the edge end.

2. The method according to claim 1, characterized in that Determining the key feature area in the feature map according to the corner point features of each feature point in the feature map includes: Dividing the feature map into at least two feature blocks; According to the corner point features of each feature point in each of the feature blocks, a key block is determined from the at least two feature blocks, and the key block is used as the key feature area.

3. The method according to claim 2, characterized in that The determining of the key block from the at least two feature blocks according to the corner point features of each feature point in each feature block comprises: Using a bilinear interpolation algorithm, according to the corner point features of each feature point in each feature block, determine the bilinear average interpolation of each feature block; The key block is determined from the at least two feature blocks according to bilinear average interpolation of each of the feature blocks.

4. The method according to claim 3, characterized in that The bilinear interpolation algorithm is used to determine the bilinear average interpolation of each feature block according to the corner point features of each feature point in each feature block, including: Using a bilinear interpolation algorithm, according to the corner point features of each feature point in each feature block, determine the bilinear interpolation of each feature point in each feature block; According to the bilinear interpolation of each feature point in each feature block, the bilinear average interpolation of each feature block is determined.

5. The method according to any one of claims 1 to 4, characterized in that The corner point features include: corner point position and corner point value.

6. The method according to claim 3, characterized in that The step of determining a key block from the at least two feature blocks according to bilinear average interpolation of each feature block comprises: Determine the number of non-critical blocks according to a preset compression ratio and the number of feature blocks; A key block is determined from the at least two feature blocks according to the number of the non-key blocks and the bilinear average interpolation of each feature block.

7. The method according to claim 6, characterized in that The determining of the key block from the at least two feature blocks according to the number of the non-key blocks and the bilinear average interpolation of each feature block comprises: Sorting the at least two feature blocks according to the bilinear average interpolation of each of the feature blocks; Selecting a non-key block from the at least two feature blocks according to the number and the sorting result of the non-key blocks; The feature blocks other than the non-key blocks in the at least two feature blocks are used as key blocks.

8. The method according to claim 7, characterized in that The step of encoding the key feature area to obtain a feature code stream includes: Encoding the key blocks in the key feature area to obtain encoded visual blocks; A feature code stream is generated according to the position embedding information of the coded visual block and the non-critical block in the feature map.

9. The method according to claim 8, characterized in that The encoding of the key blocks in the key feature area to obtain the encoded visual blocks includes: Performing linear projection on the key blocks in the key feature region to obtain block embedding information of the key blocks; The block embedding information of the key block and the position embedding information of the key block in the feature map are encoded by a visual encoder to obtain an encoded visual block corresponding to the key block.

10. The method according to claim 9, characterized in that The method further comprises: Receiving a trained visual encoder sent by the cloud; wherein the trained visual encoder is obtained by training the cloud based on a training data set updated by the received decoded visual block; and the decoded visual block is obtained by decoding the edge end based on the feature code stream; Use the trained visual encoder to replace the local visual encoder.

11. The method according to claim 1, characterized in that: The method further comprises: Receive the target visual task processing request sent by the edge end; The image acquisition device is controlled to acquire the target image to be encoded corresponding to the target visual task processing request.

12. The method according to claim 11, characterized in that The target visual task processing request is generated by the edge based on the original visual task processing request sent by the cloud; the original visual task processing request is generated by the cloud based on the task call request sent by the user.

13. A visual task processing method, characterized in that: For edge end, the method comprises: Acquire a feature code stream sent by the terminal; wherein the feature code stream is obtained by encoding a key feature area by the terminal, and the key feature area is determined by the terminal according to corner point features of each feature point in a feature map of a target image to be encoded; The feature code stream is decoded to obtain a decoded visual block.

14. The method according to claim 13, characterized in that The feature code stream is generated by the terminal according to the coded visual block obtained by encoding the key block in the key feature area, and the position embedding information of the non-key block in the feature map; Correspondingly, decoding the feature code stream to obtain a decoded visual block includes: Obtaining from the feature code stream the position embedding information of the coded visual block and the non-critical block in the feature map; The position embedding information of the coded visual block and the non-critical block in the feature map is decoded by a visual decoder to obtain a decoded visual block.

15. The method according to claim 14, characterized in that The method further comprises: Sending the decoded visual block to the cloud; wherein the decoded visual block is used for the cloud to update the training data set; Receiving a trained visual decoder sent by the cloud; wherein the trained visual decoder is trained based on an updated training data set; Use the trained visual decoder to replace the local visual decoder.

16. The method according to any one of claims 13 to 15, characterized in that The method further comprises: Receiving an original visual task processing request sent by a cloud; wherein the original visual task processing request is generated by the cloud based on a task call request sent by a user terminal; Based on the original visual task processing request, sending a target visual task processing request to the terminal; wherein the target visual task processing request is used to request the terminal to feedback the feature code stream; Accordingly, the method further includes: According to the decoded visual block, a visual task is performed to obtain a task processing result; Feedback the task processing result to the user terminal.

17. A visual task processing method, characterized in that: For use in the cloud, the method includes: Receive the task call request sent by the user; Based on the task call request, an original visual task processing request is sent to the edge end; wherein the original visual task processing request is used to instruct the edge end to send a target visual task processing request to the terminal; the target visual task processing request is used to instruct the terminal to encode the key feature area in the feature map of the target image to be encoded according to the corner point features of each feature point in the feature map to obtain a feature code stream, and the edge end determines the task processing result according to the feature code stream.

18. The method according to claim 17, characterized in that The sending the original visual task processing request to the edge end based on the task calling request includes: Acquire user information associated with the user terminal according to the task call request; Performing authority verification on the user information; If the authority verification is passed, the original visual task processing request is sent to the edge.

19. The method according to claim 17, characterized in that The method further comprises: Receiving a decoded visual block sent by the edge end; Using the decoded visual block, updating the training data set; When the updated training data set meets the training conditions, the visual encoder and / or the visual decoder are trained using the updated training data set; The trained visual decoder is sent to the edge end, and / or the trained visual encoder is sent to the terminal.

20. A visual task processing device, characterized in that: Configured in a terminal, the device includes: A feature extraction module is used to extract features of the target image to be encoded to obtain a feature map; A region determination module, used to determine a key feature region in the feature map according to corner point features of each feature point in the feature map; The encoding and transmission module is used to encode the key feature area to obtain a feature code stream, and send the feature code stream to the edge end.

21. A visual task processing device, characterized in that: Configured at the edge, the device includes: A code stream acquisition module, used to acquire a characteristic code stream sent by a terminal; wherein the characteristic code stream is obtained by encoding a key feature area by the terminal, and the key feature area is determined by the terminal according to the corner point features of each feature point in a feature map of a target image to be encoded; The decoding module is used to decode the feature code stream to obtain a decoded visual block.

22. A visual task processing device, characterized in that: Configured in the cloud, the device includes: A request receiving module, used to receive a task calling request sent by a user terminal; A request sending module is used to send an original visual task processing request to the edge end based on the task call request; the original visual task processing request is used to instruct the edge end to send a target visual task processing request to the terminal; the target visual task processing request is used to instruct the terminal to encode the key feature area in the feature map of the target image to be encoded according to the corner point features of each feature point in the feature map to obtain a feature code stream, and the edge end determines the task processing result according to the feature code stream.

23. A terminal device, characterized in that: include: a first processor and a first transmitter; The first processor is used to extract features of the target image to be encoded to obtain a feature map; determine a key feature area in the feature map according to corner point features of each feature point in the feature map; and encode the key feature area to obtain a feature code stream; The first transmitter is used to send the feature code stream to the edge end.

24. An edge device, characterized in that: include: a first receiver and a second processor; The first receiver is used to obtain a feature code stream sent by a terminal; wherein the feature code stream is obtained by encoding a key feature area by the terminal, and the key feature area is determined by the terminal according to corner point features of each feature point in a feature map of a target image to be encoded; The second processor is used to decode the feature code stream to obtain a decoded visual block.

25. A cloud device, characterized in that: including a second receiver and a second transmitter; The second receiver is used to receive a task calling request sent by a user terminal; The second transmitter is used to send an original visual task processing request to the edge end based on the task call request; wherein the original visual task processing request is used to instruct the edge end to send a target visual task processing request to the terminal; the target visual task processing request is used to instruct the terminal to encode the key feature area in the feature map of the target image to be encoded according to the corner point features of each feature point in the feature map to obtain a feature code stream, and the edge end determines the task processing result according to the feature code stream.

26. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 19 are implemented.

Images