Video encoding and decoding method, device, equipment and medium for machine vision tasks

Abstract

The present disclosure provides a video encoding and decoding method, device, equipment and medium for machine vision tasks, and relates to the field of machine vision technology. The method includes: obtaining visual task information corresponding to the video image to be encoded; judging whether the video image to be encoded meets the image preprocessing conditions before encoding according to the visual task information corresponding to the video image to be encoded; if the video image to be encoded meets the image preprocessing conditions before encoding, performing a preprocessing operation on the video image to be encoded, and encoding the video image after the preprocessing operation; if the video image to be encoded does not meet the image preprocessing conditions before encoding, directly encoding the video image to be encoded. The present disclosure reduces the redundancy caused by the difference in image size between the source and the input layer of the downstream visual task network, and retains the accuracy of subsequent machine vision tasks while reducing the redundancy.

Description

Background Art

[0002] With the growth of machine learning applications, many intelligent platforms have been adopted in the fields of Internet of Vehicles, video surveillance, smart cities, etc., which generate massive data communications between these platforms and a large number of sensors. The increase in data volume directly leads to the decrease in efficiency of previous coding methods for human vision, and it is difficult to meet the reality in terms of latency and scale. Feature coding for intelligent machines is on the agenda.

[0003] Image preprocessing generally includes: image digitization, geometric transformation, normalization, smoothing, restoration, enhancement, etc. Image enhancement is one of the most basic contents in digital image processing technology and one of the preprocessing methods. Image enhancement is divided into image enhancement based on spatial domain and image enhancement based on frequency domain. The spatial domain refers to the image plane itself. This type of method is based on direct processing of image pixels. The frequency domain is based on modifying the Fourier transform of the image. The spatial domain processing method directly processes the grayscale value of each pixel in the two-dimensional space composed of image pixels. It can be an operation between pixels within an image or between pixels between several images. The frequency domain processing method is to indirectly process the image in the transform domain of the image. Representative spatial domain image enhancement processing methods include mean filtering and median filtering, both of which can remove or reduce noise. Image enhancement technology based on frequency domain. Generally speaking, the edges and noise of the image correspond to the high-frequency part in Fourier transform, so low-pass filtering can smooth the image and remove noise. The part where the image grayscale undergoes fusion corresponds to the high-frequency component of the spectrum, so a high-pass filter is used to attenuate or suppress the low-frequency component, which can sharpen the image.

[0004] While focusing on improving the internal algorithm of the encoder, the existing solutions ignore the redundancy caused by the difference in size between the original source and the input layer of the downstream intelligent task network. The difference between the number of nodes in the input layer of the network model and the number of pixels in a single frame of the original video is the redundancy caused by the existing compression scheme. Therefore, how to eliminate redundancy while retaining the accuracy of subsequent intelligent tasks as much as possible is an urgent problem to be solved.

[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0006] The present invention provides a video encoding and decoding method, device, equipment and medium for machine vision tasks, which at least to some extent overcome the technical problem of large data transmission volume in traditional video encoding solutions.

[0007] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0008] According to one aspect of the present disclosure, a video encoding method for machine vision tasks is provided, the method comprising: obtaining visual task information corresponding to a video image to be encoded; judging whether the video image to be encoded satisfies a pre-encoding image preprocessing condition based on the visual task information corresponding to the video image to be encoded; if the video image to be encoded satisfies the pre-encoding image preprocessing condition, performing a preprocessing operation on the video image to be encoded, and encoding the video image after the preprocessing operation is performed; if the video image to be encoded does not satisfy the preprocessing condition before encoding, directly encoding the video image to be encoded.

[0009] In some embodiments, obtaining the visual task information corresponding to the video image to be encoded includes at least one of the following: before capturing the video image, requesting the visual task information corresponding to the video image to be encoded from the image decoding end; during the process of capturing the video image, requesting the visual task information corresponding to the video image to be encoded from the image decoding end.

[0010] In some embodiments, the method further comprises: generating a preprocessing flag of the video image to be encoded, wherein the preprocessing flag is used to identify whether a preprocessing operation has been performed on the video image to be encoded.

[0011] In some embodiments, the method further includes: generating a binary code stream according to the encoding result, preprocessing identifier and visual task information of the video image to be encoded, and sending it to an image decoding end, wherein the image decoding end is used to decode the received binary code stream and perform corresponding visual tasks on the decoded video image according to the preprocessing identifier and visual task information in the decoding result.

[0012] In some embodiments, the image preprocessing operation includes: adjusting size information of the video image so that the adjusted video image meets the size information required by the machine vision task model.

[0013] According to one aspect of the present disclosure, a video decoding method for machine vision tasks is also provided, the method comprising: receiving a binary code stream from an image encoding end, wherein the binary code stream is generated by the image encoding end according to the encoding result of the video image to be encoded, a preprocessing identifier and visual task information, and the preprocessing identifier is used to identify whether the video image to be encoded has been preprocessed; decoding the binary code stream from the image encoding end to obtain a decoded video image, a preprocessing identifier and visual task information; and performing corresponding visual tasks on the decoded video image according to the preprocessing identifier and visual task information obtained by decoding.

[0014] In some embodiments, performing corresponding visual tasks on the decoded video image according to the preprocessing identifier and visual task information obtained by decoding includes: judging whether the visual task to be performed is a human eye visual task according to the visual task information obtained by decoding; if the visual task to be performed is a human eye visual task, directly inputting the decoded video image into the human eye visual task model; if the visual task to be performed is a machine vision task, judging whether the decoded video image has been preprocessed according to the preprocessing identifier obtained by decoding; if the decoded video image has been preprocessed, directly inputting the decoded video image into the corresponding machine vision task model according to the visual task information obtained by decoding; if the decoded video image has not been preprocessed, performing preprocessing operations on the decoded video image according to the visual task information obtained by decoding, and inputting the video image after the preprocessing operations into the corresponding machine vision task model.

[0015] According to another aspect of the present disclosure, a video encoding device for machine vision tasks is also provided, including: a task information acquisition module, used to acquire visual task information corresponding to a video image to be encoded; an image preprocessing module, used to determine whether the video image to be encoded meets a preprocessing condition before encoding based on the visual task information corresponding to the video image to be encoded; a first image encoding module, used to perform a preprocessing operation on the video image to be encoded if the video image to be encoded meets the preprocessing condition before encoding, and encode the video image after the preprocessing operation is performed; a second image encoding module, used to directly encode the video image to be encoded if the video image to be encoded does not meet the preprocessing condition before encoding.

[0016] According to another aspect of the present disclosure, a video decoding device for machine vision tasks is also provided, including: a data receiving module, used to receive a binary code stream from an image encoding end, wherein the binary code stream is generated by the image encoding end according to the encoding result of the video image to be encoded, a preprocessing identifier and visual task information, and the preprocessing identifier is used to identify whether the video image to be encoded has performed a preprocessing operation; an image decoding module, used to decode the received binary code stream to obtain a decoded video image, a preprocessing identifier and visual task information; a task execution module, used to execute corresponding visual tasks on the decoded video image according to the preprocessing identifier and visual task information obtained by decoding.

[0017] According to another aspect of the present disclosure, an electronic device is also provided, which includes: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any one of the above-mentioned video encoding methods for machine vision tasks by executing the executable instructions.

[0018] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the video encoding method for machine vision tasks described in any one of the above is implemented.

[0019] The video encoding and decoding method, apparatus, device and medium for machine vision tasks provided by the embodiments of the present disclosure first obtain visual task information of the video image before encoding the video image at the encoding end, so as to determine whether the video image needs to be preprocessed according to the visual task information of the video image. For the video image that needs to be preprocessed, the video image is preprocessed at the encoding end, so that the video image decoded by the decoding end can be image preprocessed, and the corresponding visual task processing can be directly performed, thereby reducing the redundancy caused by the difference in image size between the source and the input layer of the downstream visual task network, and retaining the accuracy of subsequent machine vision tasks while reducing the redundancy.

[0020] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0022] Figure 1 A schematic diagram of a traditional video encoding and decoding system architecture in the related art is shown;

[0023] Figure 2 A schematic diagram of a video encoding and decoding system architecture for machine vision tasks in an embodiment of the present disclosure is shown;

[0024] Figure 3 A flow chart of a video encoding method for machine vision tasks in an embodiment of the present disclosure is shown;

[0025] Figure 4 A flow chart of a video decoding method for machine vision tasks in an embodiment of the present disclosure is shown;

[0026] Figure 5 A specific implementation flow chart of a video encoding method for machine vision tasks in an embodiment of the present disclosure is shown;

[0027] Figure 6 A schematic diagram of a video encoding device for machine vision tasks in an embodiment of the present disclosure is shown;

[0028] Figure 7 A schematic diagram of a video decoding device for machine vision tasks in an embodiment of the present disclosure is shown;

[0029] Figure 8 A structural block diagram of an electronic device in an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0030] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0031] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0032] For ease of understanding, before introducing the embodiments of the present disclosure, several terms involved in the embodiments of the present disclosure are first explained as follows:

[0033] Machine vision: It is a rapidly developing branch of artificial intelligence. Simply put, machine vision is to use machines to replace human eyes for measurement and judgment. The machine vision system uses machine vision products (i.e. image acquisition devices, divided into CMOS and CCD) to convert the captured target into image signals, transmit them to a dedicated image processing system, obtain the morphological information of the captured target, and convert them into digital signals based on pixel distribution, brightness, color and other information; the image system performs various operations on these signals to extract the characteristics of the target, and then controls the action of the equipment on site based on the judgment results.

[0034] The embodiments of the present disclosure are described in detail below in conjunction with the accompanying drawings.

[0035] Different visual tasks have different requirements for video images. For example, human vision tasks can recognize video images of any size; but machine vision tasks often require the input of video images of a fixed size. Therefore, before the captured video images are input into the machine vision task model, it is often necessary to perform image preprocessing operations on the video images, including but not limited to modifying the size information of the video images.

[0036] Figure 1 A schematic diagram of a traditional video encoding and decoding system architecture in the related art is shown. Figure 1 As shown in the figure, in order to meet the requirements of machine vision tasks, the image preprocessing operation of the traditional video encoding scheme is performed on the decoding end. Since video image encoding is to reduce the amount of data during video image transmission, the video is compressed at the encoding end; it is decompressed at the decoding end to restore the video image with original size information. In order to meet the requirements of machine vision tasks, it is sometimes necessary to reduce the video image through image preprocessing operations to obtain the fixed-size video image required by the machine vision task model. If the image preprocessing operation is performed at the decoding end, it may cause redundancy problems.

[0037] Figure 2 A schematic diagram of a video encoding and decoding system architecture for machine vision tasks in an embodiment of the present disclosure is shown. Figure 2 As shown, in the disclosed embodiment, the image preprocessing operation is moved forward to the encoding end, so that before encoding the video image, the visual task information to be performed on the video image to be encoded can be obtained, and based on the visual task information to be performed on the video image to be encoded, it can be determined whether it is necessary to perform a pre-encoding image preprocessing operation on the image to be encoded. For the video image that needs to perform a pre-encoding image preprocessing operation, the image preprocessing is performed first and then the encoding compression is performed, so that the video image decoded by the decoding end is the video image after the image preprocessing operation is performed, and the corresponding visual task can be performed. In this way, not only the problem of redundant processing can be avoided, but also the compression ratio can be improved. By Figure 2 It can be seen that for video images that meet human vision requirements and do not meet the pre-encoding image preprocessing conditions, the video encoding and decoding scheme used is consistent with the traditional video encoding scheme; for video images that meet machine vision requirements and meet the pre-encoding image preprocessing conditions, the pre-encoding image preprocessing operation is first performed, and then enters the video image encoder to form a binary code stream; the decoding end decodes the binary code stream and directly sends it to the machine vision task model. For human vision requirements, the decoded video image can be used directly, or it can be further subjected to image post-processing / image enhancement operations before entering the human vision task model.

[0038] In order to eliminate the redundancy between the original signal source and the input layer of the downstream intelligent task network and retain the accuracy of the subsequent machine vision tasks as much as possible, a video encoding method for machine vision tasks is provided in the embodiments of the present disclosure, and the method can be performed by any electronic device with computing and processing capabilities. Including but not limited to smart phones, tablet computers, laptops, desktop computers, wearable devices, augmented reality devices, virtual reality devices, etc. Optionally, the clients of the applications installed in these electronic devices are the same, or the clients of the same type of applications based on different operating systems. Based on the different terminal platforms, the specific form of the client of the application may also be different. For example, the application client can be a mobile client, a PC client, etc.

[0039] Figure 3 A flow chart of a video encoding method for machine vision tasks in an embodiment of the present disclosure is shown as follows: Figure 3 As shown, the video encoding method for machine vision tasks provided in the embodiment of the present disclosure includes the following steps:

[0040] S302: Obtain visual task information corresponding to the video image to be encoded.

[0041] It should be noted that the visual task information obtained in the above S302 includes but is not limited to: the task type of the visual task to be performed for the video image to be encoded, and the image preprocessing operation required to perform the visual task of the corresponding task type. When the video image in the embodiment of the present disclosure is shot by a camera, the task type of the visual task can be determined by the user mode User of the camera. When the user mode User is M, it indicates that the visual task to be performed is a machine vision task; when the user mode User is P, it indicates that the visual task to be performed is a human vision task.

[0042] For the captured video images, the user mode can be determined before encoding:

[0043] 1) The visual requirements of the human eye are consistent with traditional video encoding technology.

[0044] 2) For video images that meet machine vision requirements and pre-encoding image preprocessing conditions, image preprocessing operations are first performed to form video images V1 and then enter the video encoder to form a binary code stream. The decoding end uses the corresponding video decoder to decode and generate decoded video images V1'. Video image V1' directly performs machine vision tasks. For human vision requirements, video image V1' can be used directly, or it can be entered into human vision after image post-processing / image enhancement operations.

[0045] 3) For video images that meet machine vision requirements and do not meet the image preprocessing conditions before encoding, it is consistent with traditional video encoding technology.

[0046] In some embodiments, when obtaining the visual task information corresponding to the video image to be encoded, the above S302 may request the visual task information corresponding to the video image to be encoded from the image decoding end before capturing the video image. In this way, the purpose of obtaining once and using multiple times can be achieved, which is more suitable for scenarios where machine vision tasks and human vision tasks are performed according to certain rules.

[0047] In some other embodiments, when the visual task information corresponding to the video image to be encoded is obtained in the above S302, the visual task information corresponding to the video image to be encoded can also be requested from the image decoding end during the process of collecting the video image. In this way, real-time task processing can be satisfied, which is particularly suitable for the scenario where machine vision tasks and human vision tasks are cross-executed.

[0048] S304: judging whether the video image to be encoded satisfies a pre-encoding image preprocessing condition according to the visual task information corresponding to the video image to be encoded.

[0049] It should be noted that after obtaining the visual task information of the video image to be encoded, it can be determined whether the video image to be encoded needs to perform image preprocessing operations, that is, whether the image preprocessing conditions before encoding are met, based on the visual task information corresponding to the video image to be encoded.

[0050] For human eye visual tasks, image preprocessing is not required in most cases. Therefore, if the visual task to be performed on the video image to be encoded is a human eye visual task, it means that the video image to be encoded does not need to perform image preprocessing operations. Therefore, the judgment result is that the video image to be encoded does not meet the image preprocessing conditions before encoding.

[0051] For machine vision tasks, it is often required to input images with fixed size information. Before the image is input into the machine vision task model, image preprocessing is required (including but not limited to: adjusting the image height and / or width), and the image encoding process often involves adjusting the image size. Therefore, when judging whether the video image to be encoded meets the image processing conditions before encoding based on the machine vision task information, it can be determined whether the video image to be encoded meets the image preprocessing conditions before encoding based on the video image size information required by the machine vision task and the size information of the video image before encoding.

[0052] In one scenario, assuming that the original size information of a video image is the first size information, and the image size information required by the corresponding visual task is the second size information (the second size information is smaller than the first size information), according to the existing video encoding and decoding scheme, the encoding end compresses and encodes the video image of the first size information, and after decoding by the decoding end, the obtained video image is still the first size information, and it is necessary to perform image preprocessing (reducing the first size to the second size) at the decoding end before sending it to the machine vision task model. According to the video encoding and decoding scheme provided in the embodiment of the present disclosure, the encoding end knows that the size information required for the machine vision task is the second size information, and directly performs image preprocessing (reducing the first size to the second size) on the video image at the encoding end, and then only needs to encode and decode the video image of the second size information.

[0053] In another scenario, assuming that the original size information of a video image is the third size information, and the image size information required by the corresponding visual task is the fourth size information (the fourth size information is larger than the first size information), it can be seen that the image preprocessing operation required by the machine vision task is to enlarge the image size (the third size is enlarged to the fourth size). At this time, according to the video encoding and decoding scheme provided in the embodiment of the present disclosure, it is considered that the video image to be encoded does not meet the image preprocessing conditions before encoding, that is, there is no need to perform image preprocessing on the video image at the encoding end.

[0054] S306: If the video image to be encoded meets the image preprocessing condition before encoding, a preprocessing operation is performed on the video image to be encoded, and the video image after the preprocessing operation is performed is encoded.

[0055] It should be noted that in the above steps, for the video image to be encoded that meets the image processing conditions, it is indicated that an image preprocessing operation needs to be performed, and therefore, the video image needs to be preprocessed before being sent to the encoder for encoding. The image preprocessing operation in the disclosed embodiment includes but is not limited to: adjusting the size information of the video image so that the adjusted video image meets the size information required by the machine vision task model.

[0056] In some embodiments, for a video image to be encoded that meets the image preprocessing conditions before encoding, the video encoding method for machine vision tasks provided in the embodiments of the present disclosure also includes the following steps: generating a preprocessing identifier for the video image to be encoded, wherein the preprocessing identifier is used to identify whether the video image to be encoded has been preprocessed.

[0057] Furthermore, in some embodiments, after generating a preprocessing identifier of a video image to be encoded, the video encoding method for machine vision tasks provided in the embodiments of the present disclosure also includes the following steps: generating a binary code stream according to the encoding result of the video image to be encoded, the preprocessing identifier and the visual task information, and sending it to an image decoding end, wherein the image decoding end is used to decode the received binary code stream, and perform corresponding visual tasks on the decoded video image according to the preprocessing identifier and the visual task information in the decoding result.

[0058] In this embodiment, the visual task information and the encoded video image are sent together to the image decoding end, so that the image decoding end can determine the machine vision task to be performed on the decoded video image based on the visual task information in the decoding result, and then call the corresponding visual task model to process the decoded video image.

[0059] Furthermore, the preprocessing identifier and the encoded video image are sent to the image decoding end together, so that the image decoding end can determine whether the decoded video image has undergone image preprocessing operations based on the preprocessing identifier in the decoding result, so that the video image that has undergone image preprocessing operations can be directly sent to the machine vision model for processing.

[0060] S308: If the video image to be encoded does not meet the image preprocessing condition before encoding, the video image to be encoded is directly encoded.

[0061] In the above steps, for the video images to be encoded that do not meet the image processing conditions, it is indicated that no image preprocessing operation is required, and thus, the images can be directly sent to the encoder for encoding.

[0062] From the above, it can be seen that the video encoding method for machine vision tasks provided by the embodiments of the present disclosure moves the image preprocessing operation forward to the encoding end, which can reduce the redundancy caused by the difference in image size between the source and the input layer of the downstream visual task network, and retain the accuracy of subsequent machine vision tasks while reducing the redundancy.

[0063] Based on the same inventive concept, an embodiment of the present disclosure also provides a video decoding method for machine vision tasks, which can be executed by any electronic device with computing and processing capabilities.

[0064] Figure 4 A flow chart of a video decoding method for machine vision tasks in an embodiment of the present disclosure is shown as follows: Figure 4 As shown, the video decoding method for machine vision tasks provided in the embodiment of the present disclosure includes the following steps:

[0065] S402, receiving a binary code stream from an image encoding end, wherein the binary code stream is generated by the image encoding end according to an encoding result of a video image to be encoded, a preprocessing flag, and visual task information, and the preprocessing flag is used to identify whether a preprocessing operation has been performed on the video image to be encoded;

[0066] S404, decoding the binary code stream from the image encoding end to obtain a decoded video image, a preprocessing identifier, and visual task information;

[0067] S406, performing a corresponding visual task on the decoded video image according to the preprocessing identifier and visual task information obtained through decoding.

[0068] It should be noted that the image encoding end and the image decoding end in the embodiment of the present disclosure can be deployed on the same device or on different devices. Among them, the image encoding end obtains the visual task information of the video image from the image decoding end, and determines whether to perform image preprocessing operations on the video image according to the visual task information of the video image. For the encoded video image that needs to perform image preprocessing operations, the video image is encoded after performing the preprocessing operations; for the encoded video image that does not need to perform image preprocessing operations, the video image is directly encoded; finally, the encoded video image, the preprocessing identifier and the visual task information are transmitted to the image decoding end in the form of a binary code stream, so that the image decoding end decodes the received binary code stream to obtain the decoded video image, the preprocessing identifier and the visual task information, and then performs the corresponding visual task on the decoded video image according to the decoded preprocessing identifier and the visual task information.

[0069] In some embodiments, performing corresponding visual tasks on the decoded video image according to the preprocessing identifier and visual task information obtained by decoding includes: judging whether the visual task to be performed is a human eye visual task according to the visual task information obtained by decoding; if the visual task to be performed is a human eye visual task, directly inputting the decoded video image into the human eye visual task model; if the visual task to be performed is a machine vision task, judging whether the decoded video image has been preprocessed according to the preprocessing identifier obtained by decoding; if the decoded video image has been preprocessed, directly inputting the decoded video image into the corresponding machine vision task model according to the visual task information obtained by decoding; if the decoded video image has not been preprocessed, performing preprocessing operations on the decoded video image according to the visual task information obtained by decoding, and inputting the video image after the preprocessing operations into the corresponding machine vision task model.

[0070] From the above, it can be seen that the video decoding method for machine vision tasks provided by the embodiments of the present disclosure moves the image preprocessing operation forward to the encoding end, which can reduce the redundancy caused by the difference in image size between the source and the input layer of the downstream visual task network, and retain the accuracy of subsequent machine vision tasks while reducing the redundancy.

[0071] Figure 5 A specific implementation flow chart of a video encoding method for machine vision tasks in an embodiment of the present disclosure is shown as follows: Figure 5 As shown, the specific steps include:

[0072] Before the image acquisition officially starts, the encoder requests the user mode User and the image preprocessing mode Mode configuration from the decoder; or after the image acquisition starts, the encoder requests the mode User and the image preprocessing mode Mode configuration from the decoder online.

[0073] S502, for the collected video image x with a height of h and a width of w, the user mode User is judged: if the user mode is the human vision mode P, the preprocessing flag flag=0 is set, and the process goes to S506; if the user mode is the machine vision mode M, the process goes to S504.

[0074] S504, pre-process the image. If the pre-processing condition is met, set the pre-processing flag flag = 1, perform image pre-processing operation, and enter S506; if the pre-processing condition is not met, set the pre-processing flag flag = 0, and enter S506.

[0075] Optionally, there are multiple methods for image preprocessing, including but not limited to the following:

[0076] Method 1: Determine whether the image height h is greater than the height H after image preprocessing;

[0077] Method 2: Determine whether the image width w is greater than the width W after image preprocessing;

[0078] Method 3: Determine whether the product of the image height h and the image width w (h×w) is greater than the product of the image preprocessing height W and the image preprocessing width W (H×W);

[0079] Optionally, the image preprocessing operation may also include multiple methods, such as a ResNet inference image preprocessing method, a MobileNet inference image preprocessing method, and the like.

[0080] S506, encode the image entering the encoder, and form a binary code stream with the encoded video image output by the encoder and (User, flag, h, w).

[0081] S508, the decoding end decodes the received binary code stream to form a decoded video image x_hat and (User, flag, h, w).

[0082] S510, determine the user mode User: if the user mode is the human vision mode P, directly perform human vision processing, or enter the human vision model after image post-processing / image enhancement and other processing; if the user mode is the machine vision model M, enter S512.

[0083] S512, judge the preprocessing flag flag. If the preprocessing flag is 1, directly enter the machine vision model processing, where h and w are used to locate the target position in the original image when obtaining the target position information; if the preprocessing flag is 0, perform image preprocessing operations on the decoded video image x_hat first, and then enter the machine vision model processing.

[0084] Next, let's take a practical example:

[0085] Taking the traditional 2k resolution video image collected for target detection task as an example, the target detection network model is Faster R-CNN X101 FPN, and the specific implementation is as follows:

[0086] Before officially starting image acquisition, the encoder requests the decoder to configure the user mode User and image preprocessing mode Mode, and sets the user mode User to M and the image preprocessing mode Mode to ResNetX101.

[0087] 1) For the acquired traditional 2k resolution video image x, perform mode User judgment, and if mode User is the machine vision mode M, proceed to step 2);

[0088] 2) Preprocess the video image x: the image height h is 1920, the image width is 1080, the image height H after ResNetX101 image preprocessing is 1333, the image width W is 750, the product of the image height h and the image width w is greater than the product of the image height W after preprocessing and the image width W after preprocessing, the preprocessing flag flag = 1, and the image data before entering the encoder is reduced to 48.2% of the original;

[0089] 3) Encode the image 1333x750 entering the encoder, using VTM8.2 for encoding, or an end-to-end neural network encoding method, and form a binary code stream with the encoder output and (M, 1, 1920, 1080);

[0090] 4) The decoding end decodes the received binary code stream to form a decoded video image x_hat, (M, 1, 1920, 1080);

[0091] 5) Judge the mode User, under the machine vision model M, and go to step 6);

[0092] 6) The preprocessing flag is used for judgment. If the preprocessing flag is 1, the image directly enters the machine vision model processing, where h and w are used for positioning in the original image when obtaining the target position information.

[0093] Based on the same inventive concept, the present disclosure also provides a video encoding device for machine vision tasks, as described in the following embodiments. Since the principle of solving the problem in the device embodiment is similar to that in the above method embodiment, the implementation of the device embodiment can refer to the implementation of the above method embodiment, and the repeated parts will not be repeated.

[0094] Figure 6 A schematic diagram of a video encoding device for machine vision tasks in an embodiment of the present disclosure is shown. Figure 6 As shown, the device includes: a task information acquisition module 61, an image preprocessing module 62, a first image encoding module 63 and a second image encoding module 64.

[0095] Among them, the task information acquisition module 61 is used to obtain the visual task information corresponding to the video image to be encoded; the image preprocessing module 62 is used to determine whether the video image to be encoded meets the image preprocessing conditions before encoding based on the visual task information corresponding to the video image to be encoded; the first image encoding module 63 is used to perform a preprocessing operation on the video image to be encoded if the video image to be encoded meets the image preprocessing conditions before encoding, and encode the video image after the preprocessing operation is performed; the second image encoding module 64 is used to directly encode the video image to be encoded if the video image to be encoded does not meet the image preprocessing conditions before encoding.

[0096] It should be noted that the task information acquisition module 61, the image preprocessing module 62, the first image encoding module 63 and the second image encoding module 64 correspond to S302 to S308 in the method embodiment, and the examples and application scenarios implemented by the modules and the corresponding steps are the same, but are not limited to the contents disclosed in the method embodiment. It should be noted that the modules as part of the device can be executed in a computer system such as a set of computer executable instructions.

[0097] From the above, it can be seen that the video encoding device for machine vision tasks provided by the embodiments of the present disclosure moves the image preprocessing operation forward to the encoding end, which can reduce the redundancy caused by the difference in image size between the source and the input layer of the downstream visual task network, and retain the accuracy of subsequent machine vision tasks while reducing the redundancy.

[0098] In some embodiments, obtaining visual task information corresponding to the video image to be encoded includes at least one of the following: before capturing the video image, requesting the visual task information corresponding to the video image to be encoded from the image decoding end; during the process of capturing the video image, requesting the visual task information corresponding to the video image to be encoded from the image decoding end.

[0099] In some embodiments, the method further includes: generating a preprocessing flag of the video image to be encoded, wherein the preprocessing flag is used to identify whether a preprocessing operation has been performed on the video image to be encoded.

[0100] In some embodiments, the method also includes: generating a binary code stream based on the encoding result, preprocessing identifier and visual task information of the video image to be encoded, and sending it to the image decoding end, wherein the image decoding end is used to decode the received binary code stream and perform corresponding visual tasks on the decoded video image based on the preprocessing identifier and visual task information in the decoding result.

[0101] In some embodiments, the image preprocessing operation includes: adjusting the size information of the video image so that the adjusted video image meets the size information required by the machine vision task model.

[0102] Based on the same inventive concept, the present disclosure also provides a video decoding device for machine vision tasks, as described in the following embodiments. Since the principle of solving the problem in the device embodiment is similar to that in the above method embodiment, the implementation of the device embodiment can refer to the implementation of the above method embodiment, and the repeated parts will not be repeated.

[0103] Figure 7 A schematic diagram of a video decoding device for machine vision tasks in an embodiment of the present disclosure is shown. Figure 7 As shown, the device includes: a data receiving module 71, an image decoding module 72 and a task execution module 73.

[0104] Among them, the data receiving module 71 is used to receive the binary code stream from the image encoding end, wherein the binary code stream is generated by the image encoding end according to the encoding result, preprocessing flag and visual task information of the video image to be encoded, and the preprocessing flag is used to identify whether the video image to be encoded has performed the preprocessing operation; the image decoding module 72 is used to decode the received binary code stream to obtain the decoded video image, preprocessing flag and visual task information; the task execution module 73 is used to execute the corresponding visual task on the decoded video image according to the preprocessing flag and visual task information obtained by decoding.

[0105] It should be noted that the above data receiving module 71, image decoding module 72 and task execution module 73 correspond to S402 to S406 in the method embodiment, and the examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the contents disclosed in the above method embodiment. It should be noted that the above modules as part of the device can be executed in a computer system such as a set of computer executable instructions.

[0106] From the above, it can be seen that the video decoding device for machine vision tasks provided by the embodiments of the present disclosure moves the image preprocessing operation forward to the encoding end, which can reduce the redundancy caused by the difference in image size between the source and the input layer of the downstream visual task network, and retain the accuracy of subsequent machine vision tasks while reducing the redundancy.

[0107] In some embodiments, performing corresponding visual tasks on the decoded video image according to the preprocessing identifier and visual task information obtained by decoding includes: judging whether the visual task to be performed is a human eye visual task according to the visual task information obtained by decoding; if the visual task to be performed is a human eye visual task, directly inputting the decoded video image into the human eye visual task model; if the visual task to be performed is a machine vision task, judging whether the decoded video image has been preprocessed according to the preprocessing identifier obtained by decoding; if the decoded video image has been preprocessed, directly inputting the decoded video image into the corresponding machine vision task model according to the visual task information obtained by decoding; if the decoded video image has not been preprocessed, performing preprocessing operations on the decoded video image according to the visual task information obtained by decoding, and inputting the video image after the preprocessing operations into the corresponding machine vision task model.

[0108] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as "circuits", "modules" or "systems".

[0109] Refer to the following Figure 8 The electronic device 800 according to this embodiment of the present disclosure is described. Figure 8 The electronic device 800 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0110] like Figure 8As shown, the electronic device 800 is in the form of a general computing device. The components of the electronic device 800 may include but are not limited to: at least one processing unit 810, at least one storage unit 820, and a bus 830 connecting different system components (including the storage unit 820 and the processing unit 810).

[0111] The storage unit stores a program code, and the program code can be executed by the processing unit 810, so that the processing unit 810 executes the steps described in the above "Exemplary Method" section of this specification according to various exemplary embodiments of the present disclosure. For example, the processing unit 810 can execute the following steps of the above method embodiment: obtaining visual task information corresponding to the video image to be encoded; judging whether the video image to be encoded meets the image preprocessing condition before encoding according to the visual task information corresponding to the video image to be encoded; if the video image to be encoded meets the image preprocessing condition before encoding, performing a preprocessing operation on the video image to be encoded, and encoding the video image after the preprocessing operation; if the video image to be encoded does not meet the image preprocessing condition before encoding, directly encoding the video image to be encoded.

[0112] The storage unit 820 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 8201 and / or a cache memory unit 8202 , and may further include a read-only memory unit (ROM) 8203 .

[0113] The storage unit 820 may also include a program / utility 8204 having a set (at least one) of program modules 8205, such program modules 8205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0114] Bus 830 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0115] The electronic device 800 may also communicate with one or more external devices 840 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 800, and / or communicate with any device that enables the electronic device 800 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 850. Furthermore, the electronic device 800 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 860. As shown, the network adapter 860 communicates with other modules of the electronic device 800 via a bus 830. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 800, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0116] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0117] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, which may be a readable signal medium or a readable storage medium. A program product capable of implementing the above method of the present disclosure is stored thereon. In some possible implementations, various aspects of the present disclosure may also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to cause the terminal device to execute the steps according to various exemplary implementations of the present disclosure described in the above “Exemplary Method” section of this specification.

[0118] More specific examples of computer-readable storage media in the present disclosure may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0119] In the present disclosure, a computer readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, wherein a readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0120] Alternatively, the program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0121] In a specific implementation, the program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0122] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0123] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0124] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0125] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.

Claims

1. A video encoding method for machine vision tasks, characterized in that: include: Obtaining visual task information corresponding to the video image to be encoded; Determining whether the video image to be encoded satisfies a pre-encoding image preprocessing condition according to the visual task information corresponding to the video image to be encoded; If the video image to be encoded meets the image preprocessing condition before encoding, a preprocessing operation is performed on the video image to be encoded, and the video image after the preprocessing operation is performed is encoded; If the video image to be encoded does not meet the image preprocessing condition before encoding, directly encoding the video image to be encoded; The visual task information includes: the task type of the visual task to be performed on the video image to be encoded, and the image preprocessing operation required to perform the visual task of the corresponding task type; the preprocessing operation includes: adjusting the size information of the video image so that the adjusted video image meets the size information required by the machine vision task model; Wherein, the method further comprises: generating a preprocessing flag of the video image to be encoded, wherein the preprocessing flag is used to identify whether a preprocessing operation has been performed on the video image to be encoded; A binary code stream is generated according to the encoding result, preprocessing identifier and visual task information of the video image to be encoded, and sent to an image decoding end, wherein the image decoding end is used to decode the received binary code stream, and perform corresponding visual tasks on the decoded video image according to the preprocessing identifier and visual task information in the decoding result through the following steps: judging whether the visual task to be performed is a human eye visual task according to the visual task information obtained by decoding; if the visual task to be performed is a human eye visual task, directly inputting the decoded video image into the human eye visual task model; if the visual task to be performed is a machine vision task, judging whether the decoded video image has been preprocessed according to the preprocessing identifier obtained by decoding; if the decoded video image has been preprocessed, directly inputting the decoded video image into the corresponding machine vision task model according to the visual task information obtained by decoding; if the decoded video image has not been preprocessed, performing preprocessing operations on the decoded video image according to the visual task information obtained by decoding, and inputting the video image after the preprocessing operations into the corresponding machine vision task model.

2. The video encoding method for machine vision tasks according to claim 1, characterized in that: The obtaining of visual task information corresponding to the video image to be encoded includes at least one of the following: Before capturing a video image, requesting visual task information corresponding to the video image to be encoded from an image decoding end; During the process of acquiring the video image, the image decoding end is requested for the visual task information corresponding to the video image to be encoded.

3. A video decoding method for machine vision tasks, characterized in that: include: Receiving a binary code stream from an image encoding end, wherein the binary code stream is generated by the image encoding end according to an encoding result of a video image to be encoded, a preprocessing flag, and visual task information, the preprocessing flag is used to identify whether a preprocessing operation has been performed on the video image to be encoded, and the preprocessing flag is generated by the image encoding end; Decode the binary code stream from the image encoding end to obtain the decoded video image, preprocessing mark and visual task information; According to the preprocessing identifier and visual task information obtained by decoding, a corresponding visual task is performed on the decoded video image; The visual task information includes: the task type of the visual task to be performed on the video image to be encoded, and the image preprocessing operation required to perform the visual task of the corresponding task type; the preprocessing operation includes: adjusting the size information of the video image so that the adjusted video image meets the size information required by the machine vision task model; Among them, the performing of the corresponding visual task on the decoded video image according to the preprocessing identifier and visual task information obtained by decoding includes: judging whether the visual task to be performed is a human eye visual task according to the visual task information obtained by decoding; if the visual task to be performed is a human eye visual task, directly inputting the decoded video image into the human eye visual task model; if the visual task to be performed is a machine vision task, judging whether the decoded video image has been preprocessed according to the preprocessing identifier obtained by decoding; if the decoded video image has been preprocessed, directly inputting the decoded video image into the corresponding machine vision task model according to the visual task information obtained by decoding; if the decoded video image has not been preprocessed, performing a preprocessing operation on the decoded video image according to the visual task information obtained by decoding, and inputting the video image after the preprocessing operation into the corresponding machine vision task model.

4. A video encoding device for machine vision tasks, characterized in that: include: A task information acquisition module, used to acquire visual task information corresponding to the video image to be encoded; An image preprocessing module, used to determine whether the video image to be encoded meets the image preprocessing condition before encoding according to the visual task information corresponding to the video image to be encoded; A first image encoding module, configured to perform a preprocessing operation on the video image to be encoded if the video image to be encoded meets a preprocessing condition before encoding, and encode the video image after the preprocessing operation is performed; A second image encoding module, configured to directly encode the video image to be encoded if the video image to be encoded does not meet the image preprocessing condition before encoding; The visual task information includes: the task type of the visual task to be performed on the video image to be encoded, and the image preprocessing operation required to perform the visual task of the corresponding task type; the preprocessing operation includes: adjusting the size information of the video image so that the adjusted video image meets the size information required by the machine vision task model; The video encoding device is further used to: generate a preprocessing identifier for the video image to be encoded, wherein the preprocessing identifier is used to identify whether the video image to be encoded has been preprocessed; generate a binary code stream according to the encoding result, preprocessing identifier and visual task information of the video image to be encoded, and send it to the image decoding end, wherein the image decoding end is used to decode the received binary code stream, and perform corresponding visual tasks on the decoded video image according to the preprocessing identifier and visual task information in the decoding result through the following steps: determine whether the visual task to be performed is a human eye visual task according to the visual task information obtained by decoding; if the visual task to be performed If it is a human eye vision task, the decoded video image is directly input into the human eye vision task model; if the vision task to be executed is a machine vision task, it is determined whether the decoded video image has been preprocessed according to the preprocessing identifier obtained by decoding; if the decoded video image has been preprocessed, the decoded video image is directly input into the corresponding machine vision task model according to the visual task information obtained by decoding; if the decoded video image has not been preprocessed, the decoded video image is preprocessed according to the visual task information obtained by decoding, and the video image after the preprocessing operation is input into the corresponding machine vision task model.

5. A video decoding device for machine vision tasks, characterized in that: include: A data receiving module, used for receiving a binary code stream from an image encoding end, wherein the binary code stream is generated by the image encoding end according to an encoding result of a video image to be encoded, a preprocessing flag and visual task information, and the preprocessing flag is used to identify whether a preprocessing operation has been performed on the video image to be encoded; An image decoding module is used to decode the received binary code stream to obtain a decoded video image, a preprocessing identifier and visual task information; A task execution module, used to execute corresponding visual tasks on the decoded video image according to the pre-processing identifier and visual task information obtained by decoding; The visual task information includes: the task type of the visual task to be performed on the video image to be encoded, and the image preprocessing operation required to perform the visual task of the corresponding task type; the preprocessing operation includes: adjusting the size information of the video image so that the adjusted video image meets the size information required by the machine vision task model; Among them, the task execution module is also used to: determine whether the visual task to be executed is a human eye visual task based on the visual task information obtained by decoding; if the visual task to be executed is a human eye visual task, directly input the decoded video image into the human eye visual task model; if the visual task to be executed is a machine vision task, determine whether the decoded video image has been preprocessed according to the preprocessing identifier obtained by decoding; if the decoded video image has been preprocessed, directly input the decoded video image into the corresponding machine vision task model according to the visual task information obtained by decoding; if the decoded video image has not been preprocessed, perform a preprocessing operation on the decoded video image according to the visual task information obtained by decoding, and input the video image after the preprocessing operation into the corresponding machine vision task model.

6. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to execute the video encoding method for machine vision tasks according to claim 1 or 2, or the video decoding method for machine vision tasks according to claim 3, by executing the executable instructions.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the video encoding method for machine vision tasks described in any one of claims 1 or 2, or the video decoding method for machine vision tasks described in claim 3 is implemented.

Images