Method, apparatus, and related devices for processing visual image data

Abstract

The present disclosure provides a method, apparatus, electronic device, and medium for processing visual image data. The method includes extracting features of images in the collected video to obtain image feature information, generating an image feature code stream based on the image feature information, transmitting the image feature code stream to a cloud server, receiving position information transmitted from the cloud server, where the position information is generated by the cloud server based on the image feature code stream, extracting a region image based on the position information, generating an image code stream based on the region image, and transmitting the image code stream to the cloud server. The present disclosure increases the compression ratio of the video, reduces the pressure caused by the bandwidth, enables the terminal device and the cloud server to process images simultaneously, reduces the computing amount of the cloud server, and improves the real-time processing ability.

Description

Technical Field

[0001] (Cross - reference to related applications) This disclosure claims the priority of a Chinese patent application with the application number 202210774919.0 and the title "Method, apparatus, electronic device, and medium for processing visual image data" filed on July 1, 2022, and all the contents of the Chinese patent application are incorporated herein by reference.

[0002] This disclosure relates to the field of image processing technology, and in particular, to a method, apparatus, electronic device, computer - readable storage, medium, and computer program product for processing visual image data.

Background Art

[0003] With the rapid development of artificial intelligence technology, scenes such as smart cities, smart transportation, and unmanned patrol inspections have started large - scale applications. When applying, it is necessary to collect visual images. Terminal devices collect visual images and transmit the collected visual images to a cloud server. The collected visual images often need to meet the characteristics of high pixels, which brings great pressure on the transmission bandwidth of visual images. In addition, the collected visual images are generally processed for data at the cloud server, and the real - time processing ability is not good.

[0004] It should be noted that the information disclosed in the above background art part is only used to enhance the understanding of the background of this disclosure, and may include information that does not constitute prior art known to those skilled in the art.

Summary of the Invention

[0005] This disclosure provides a method, apparatus, electronic device, medium, and computer program product for processing visual image data, which at least to some extent overcomes the problems of large pressure on the visual image transmission bandwidth and poor real - time processing ability.

[0006] Other features and advantages of the present disclosure will become apparent from the following detailed description or will be partially learned through the practice of the present disclosure.

[0007] One aspect of the present disclosure provides a method for processing visual image data applied to a terminal device. The method includes extracting features of an image in an acquired video to obtain image feature information, generating an image feature code stream based on the image feature information and transmitting it to a cloud server, receiving position information transmitted by the cloud server, where the position information is generated by the cloud server based on the image feature code stream, extracting a region image based on the position information, and generating an image code stream based on the region image and transmitting it to the cloud server.

[0008] In one embodiment of the present disclosure, extracting features of an image in the acquired video to obtain image feature information, generating an image feature code stream based on the image feature information and transmitting it to a cloud server includes extracting features of the image based on some layers of a first neural network model to generate image feature information, and feature-encoding the image feature information to generate an image feature code stream.

[0009] In one embodiment of the present disclosure, the fact that the position information is generated by the cloud server based on the image feature code stream includes the cloud server decrypting the image feature code stream to obtain decrypted image features, and inputting the decrypted image features based on the remaining layers of the first neural network model to output a first prediction result. The first prediction result includes a first predicted target category and position information.

[0010] In one embodiment of the present disclosure, after generating an image code stream based on the region image and transmitting it to the cloud server, the method further includes receiving new position information, where the new position information is included in a second prediction result, the second prediction result further includes a preset second prediction target category, the second prediction result is generated by the cloud server based on the decoded image and a second neural network model, extracting a region image based on the new position information, generating an image code stream based on the region image, and transmitting it to the cloud server.

[0011] In one embodiment of the present disclosure, after generating an image code stream based on the region image and transmitting it to the cloud server, the method further includes receiving a request for obtaining an original image transmitted by the cloud server, where the original image includes a detection target.

[0012] Another aspect of the present disclosure provides a method for processing visual image data applied to a cloud server. The method includes receiving an image feature code stream transmitted by a terminal device, where the image feature code stream is generated based on image feature information, and the image feature information is obtained by extracting features from an image in a video acquired by the terminal device, generating position information based on the image feature code stream and transmitting it to the terminal device, and receiving an image code stream transmitted by the terminal device, where the image code stream is generated by the terminal device based on extracting a region image based on the position information and based on the region image.

[0013] In one embodiment of the present disclosure, generating position information based on the image feature code stream and transmitting it to the terminal device includes decoding the image feature code stream to obtain decoded image features, and inputting the decoded image features based on a residual layer of a first neural network model to output a first prediction result, where the first prediction result includes a first prediction target category and position information.

[0014] In one embodiment of the present disclosure, generating position information based on the image feature code stream and transmitting it to the terminal device includes querying whether the first target category among the N sequentially set target categories is included in the first prediction target category, where N is a natural number, ending the process when the first target category is not included, and transmitting the position information to the terminal device when the first target category is included.

[0015] In one embodiment of the present disclosure, after receiving the image code stream transmitted by the terminal device, the method further includes confirming whether the region image includes a detection target based on the image code stream.

[0016] In one embodiment of the present disclosure, confirming whether the region image includes a detection target based on the image code stream includes decrypting the image code stream to obtain a decrypted image, inputting the decrypted image based on a second neural network model to output a second prediction result, where the second prediction result includes a second prediction target category and new position information, determining whether the M-th target category among the N sequentially set target categories is included in the second prediction target category, where M is the number of times the second prediction target category is generated plus 1, M is less than or equal to N, ending the process when the M-th target category is not included, and determining whether the second prediction target category is the same as the M-th target category when the M-th target category is included.

[0017] In one embodiment of the present disclosure, determining whether the second prediction target category is the same as the M-th target category includes when the second prediction target category is the same as the M-th target category, determining whether the second prediction target category is the same as the detection target, When the second predicted target category is the same as the detection target, output a prompt to find the detection target. When the second predicted target category is not the same as the detection target, end the process. When the second predicted target category is not the same as the Mth target category, include sending the new location information to the terminal device so that the terminal device extracts a region image based on the new location information.

[0018] In one embodiment of the present disclosure, the method further includes, when the second predicted target category is the same as the detection target, sending a request to obtain the original image to the terminal device.

[0019] Another aspect of the present disclosure provides a processing device for machine vision image data applied to a terminal device. The device includes a feature extraction and encoding module, a terminal reception module, an image extraction module, and a processing module. The feature extraction and encoding module is configured to extract features from an image in the acquired video, obtain image feature information, generate an image feature code stream based on the image feature information, and send it to a cloud server. The terminal reception module is configured to receive the location information sent by the cloud server, where the location information is generated by the cloud server based on the image feature code stream. The image extraction module is configured to extract a region image based on the location information. The processing module is configured to generate an image code stream based on the region image and send it to the cloud server.

[0020] Other aspects of the present disclosure provide an apparatus for processing machine vision image data applied to a cloud server. The apparatus includes a cloud receiving module and a position generating module. The cloud receiving module is configured to receive an image feature code stream transmitted by a terminal device, where the image feature code stream is generated based on image feature information, and the image feature information is obtained by extracting features of images in a video acquired by the terminal device. The position generating module is configured to generate position information based on the image feature code stream and transmit it to the terminal device. The cloud receiving module is further configured to receive an image code stream transmitted by the terminal device, where the image code stream is generated based on a region image extracted by the terminal device based on the position information.

[0021] Other aspects of the present disclosure provide an electronic device including a processor and a memory for storing executable instructions of the processor. The processor is configured to execute the above-described method for processing visual image data by executing the executable instructions.

[0022] Other aspects of the present disclosure provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for processing visual image data.

[0023] Other aspects of the present disclosure provide a computer program product including a computer program or computer instructions. The computer program or computer instructions are loaded and executed by a processor to cause a computer to implement any one of the above-described methods for processing visual image data.

[0024] According to the method, device, electronic device, medium, and computer program product for processing visual image data in the embodiments of the present disclosure, the terminal device extracts features of images in the collected video to obtain image feature information, generates an image feature code stream based on the image feature information, the cloud server generates location information based on the image feature code stream, the terminal device extracts a region image based on the location information, generates an image code stream based on the region image, and transmits it to the cloud server. By extracting features of images by the terminal device and generating an image feature code stream based on the image feature information, the compression rate of the video is increased, the pressure caused by the bandwidth is reduced, the terminal device and the cloud server process images simultaneously, the computing amount of the cloud server is reduced, and the real-time processing ability is enhanced.

[0025] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and do not limit the present disclosure.

Brief Description of the Drawings

[0026] Here, the drawings are incorporated into the specification, showing embodiments that conform to the present disclosure, and are used together with the specification to explain the principles of the present disclosure. Obviously, the drawings in the following description are only some embodiments of the present disclosure, and those skilled in the art can also obtain other drawings based on these drawings without creative efforts.

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Embodiments for Carrying Out the Invention

[0027] Next, with reference to the drawings, exemplary embodiments will be described in more detail. However, the exemplary embodiments can be implemented in various forms and should not be understood as being limited to the examples described herein. In contrast, these embodiments are provided to make the present disclosure more comprehensive and complete and to fully convey the concept of the exemplary embodiments to those skilled in the art. The features, structures, or characteristics described can be incorporated into one or more embodiments in any suitable manner.

[0028] Also, the drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures indicate the same or similar parts, so duplicate descriptions are omitted. Some of the block diagrams shown in the drawings are functional entities and do not necessarily need to correspond to physically or logically independent entities. These functional entities may be implemented in software form, or may be implemented by one or more hardware modules or integrated circuits, or may be implemented by different networks and / or processor devices and / or microcontroller devices.

[0029] Hereinafter, specific embodiments of the embodiments of the present disclosure will be described in detail with reference to the drawings.

[0030] FIG. 1 shows a schematic diagram of an exemplary application system architecture to which the method for processing visual image data in an embodiment of the present disclosure can be applied.

[0031] As shown in FIG. 1, this system architecture includes a terminal device including a video collection device 101 for collecting video images and an image preprocessing device 102, and a cloud server 103. The video collection device 101 may be a drone, a camera, a mobile phone, or a device having other types of video or image shooting functions. The video collection device 101 is used to collect videos or images. When the video collection device 101 has image preprocessing performance, the video collection device 101 is connected to the cloud server 103 via a network, preprocesses the collected video or image, and transfers it to the cloud server 103. The cloud server 103 is used to process the vision or image after being processed by the video collection device 101. When the video collection device 101 does not have image preprocessing performance, the video collection device 101 is connected to the cloud server 103 via an image preprocessing device 102 for preprocessing the collected video or image. The image preprocessing device 102 may be a server that provides various services, such as a background management server that supports the video collection device 101.

[0032] A person skilled in the art can know that the numbers of the video collection device 101, the image preprocessing device 102, and the cloud server 103 in FIG. 1 are merely schematic, and any number of video collection devices 101, image preprocessing devices 102, and cloud servers 103 can be provided according to actual needs. The embodiments of the present disclosure are not limited thereto.

[0033] The network for connecting the image preprocessing device 102 and the cloud server 103, and the network for connecting the video collection device 101 and the cloud server 103 may be a wired network or a wireless network.

[0034] In some embodiments, the wireless network or wired network described above uses standard communication technologies and / or protocols. The network is typically the Internet, but can be any network including, but not limited to, a Local Area Network (LAN), a Metropolitan Area Network (MAN), a Wide Area Network (WAN), a mobile, wired or wireless network, a private network, or any combination of virtual private networks. In some embodiments, technologies and / or formats including Hyper Text Mark-up Language (HTML), Extensible Markup Language (XML), etc. are used to represent data exchanged over the network. Also, normal encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPsec), etc. can be used to encrypt all or some of the links. In another embodiment, instead of the above data communication technologies, customized and / or proprietary data communication technologies can be used or added.

[0035] The cloud server 103 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain services, security services, CDN (Content Delivery Network), and infrastructure cloud computing services such as big data and artificial intelligence platforms.

[0036] In an embodiment of the present disclosure, a method for processing visual image data is provided. The terminal device extracts features of an image in the acquired video to obtain image feature information, generates an image feature code stream based on the image feature information, and transmits it to the cloud server 103. The cloud server 103 generates location information from the image feature code stream. The terminal device extracts a region image from the location information, generates an image code stream based on the region image, and transmits it to the cloud server 103. This method can be applied to scenes such as power line inspection, base station inspection, disaster relief, field scientific experiments, farm monitoring, and movie shooting by drones, and can process the collected videos and images, thereby meeting people's shooting purposes.

[0037] In some embodiments, the method for processing visual image data in the embodiments of the present disclosure can be executed on the terminal device and the cloud server 103 shown in FIG. 1.

[0038] As shown in FIG. 2, a flowchart of the method for processing visual image data in the embodiments of the present disclosure is shown, which is applied to a terminal device and includes the following steps.

[0039] In S202, extract features of an image in the acquired video to obtain image feature information, generate an image feature code stream from the image feature information, and transmit it to the cloud server.

[0040] The terminal device can collect videos or images. If the collected data is a video, the terminal device can extract frames from the collected video and write the extracted images into an image queue. The terminal device can extract features from the images through a neural network model to obtain image feature information, encode the image feature information, and generate an image feature code stream. When the image is transmitted, only the encoded image feature information is transmitted, reducing the transmission data volume of the visual image and reducing the bandwidth pressure caused by visual image transmission.

[0041] In S204, receive the position information sent by the cloud server, where the position information is generated by the cloud server based on the image feature code stream.

[0042] The position information is the position information of the region image to be extracted. The position information is a set of several coordinate points, and the position of the region image can be determined based on the multiple coordinate points. The position information is also a combination of several coordinate points and image edge information, and the position of the region image can be determined based on the multiple coordinate points and the image edge information. The image edge information can be the width of the image or the length of the image. The position of the region image can be determined by using the combination of the coordinate points and the width of the image or the combination of the coordinate points and the length of the image.

[0043] In S206, extract the region image based on the position information.

[0044] After receiving the position information sent from the cloud server, the terminal device extracts the region image based on the position information. The size of the region image is smaller than the size of the image and can be 5% - 50% of the image area size. The detection target is included in the region image. The detection target is located in the image and is larger than the region image, which is beneficial for accurately positioning and collecting the detection target. The detection target is the target image that needs to be extracted. At the same time, the size of the region image is smaller than the size of the image, which can reduce the transmission data volume of the visual image and further reduce the bandwidth pressure caused by image transmission.

[0045] In S208, an image code stream is generated based on the region image and sent to the cloud server.

[0046] The terminal device encodes the region image to generate an image code stream, and the cloud server decodes the image code stream and determines whether the decoded region image is the same as the detection target. By continuously interacting with each other between the terminal device and the cloud server, the necessary results can be obtained. For example, in the unmanned inspection project of the power grid, the vibration damper is damaged or complete results can be obtained.

[0047] In the related art, for machine vision tasks for small targets, it is often necessary to define the target area many times and perform cascade machine vision tasks. In these scenes, image collection often needs to meet high-pixel features, which brings pressure on the video transmission due to a large bandwidth. In addition, the cloud server needs to perform arithmetic processing for video transmission, and the arithmetic ability of the cloud server is limited. When the amount of data to be processed is large, the real-time processing ability is weakened, affecting the work efficiency.

[0048] The present disclosure extracts features of an image on a terminal device, generates an image code stream based on the image feature information, thereby increasing the compression ratio of the video, reducing the pressure due to the bandwidth, and the terminal device and the cloud server simultaneously cooperate to process the image, reducing the arithmetic amount of the cloud server and enhancing the real-time processing ability.

[0049] In some embodiments, extracting features of an image in the acquired video, obtaining image feature information, generating an image feature code stream based on the image feature information, and sending it to the cloud server include: extracting features of the image based on some layers of the first neural network model to generate image feature information; feature encoding the image feature information to generate an image feature code stream.

[0050] For the first neural network model, it may be a deep convolutional neural network DCNN model, a generative adversarial network GAN model, a CascadeR-CNN model, or other models capable of performing image feature extraction.

[0051] For example, the terminal device extracts features of an image through the stem layer of resneXt101 of the CascadeRCNN ResneXt 101FPN model, generates image feature information, encodes the image feature information through a feature encoding method, generates an image feature code stream, and the feature encoding method can be one of all-frame (AllIntra, AI) encoding, clustering quantization, and entropy encoding.

[0052] In some embodiments, it is that the position information is generated by the cloud server based on the image feature code stream. The cloud server decodes the image feature code stream and obtains the decoded image features, using the decoded image features as input based on the residual layer of the first neural network model to output a first prediction result, where the first prediction result includes a first predicted target category and position information.

[0053] The first predicted target category is preset in the cloud server, and the position information is used to determine the position of the region image, facilitating the terminal device to extract the region image based on the position information.

[0054] For example, using the decoded image features as input, input them into the network layer after the stem layer of the CascadeRCNN ResneXt 101FPN model to output a first prediction result, where the first prediction result includes a first predicted target category and position information, and the position information can be expressed as a coordinate set.

[0055] In some embodiments, extracting features from the images in the acquired video, obtaining image feature information, generating an image feature code stream based on the image feature information, and transmitting it to a cloud server is extracting frames from the collected video to generate images, writing the images to an image queue, and extracting features from the images written to the image queue.

[0056] For generating images, the terminal device collects a video, extracts frames from the video to generate a plurality of images, writes the images to an image queue, sequentially takes out the images located in the image queue from the stack, and the terminal device extracts features from the images taken out from the stack.

[0057] In some embodiments, after generating an image code stream based on the region image and transmitting it to a cloud server, the method further includes receiving new position information, where the new position information is included in a second prediction result, the second prediction result includes a preset second prediction target category, and the second prediction result is also generated by the cloud server based on the decoded image and a second neural network model, extracting a region image based on the new position information, generating an image code stream based on the region image, and transmitting it to a cloud server.

[0058] The cloud server decodes the image code stream, obtains a decoded image, uses the decoded image as the input of a second neural network model, outputs it to the second prediction result, and the new position information is included in the second prediction result.

[0059] In some embodiments, after generating an image code stream based on the region image and transmitting it to a cloud server, the method further includes receiving a request for obtaining an original image transmitted by the cloud server, where the original image includes a detection target.

[0060] In addition, when the second prediction target category is the same as the detection target, the cloud server sends a request to the terminal device to obtain the original image in order to check or display the original image.

[0061] In the embodiments of the present disclosure, based on the same inventive concept, a method for processing visual image data is also provided as described in the following embodiments. Since the principle of problem-solving according to the embodiments of this method is similar to that of the embodiments of the above method, the implementation of the embodiments of this method can refer to the implementation of the embodiments of the above method, and the overlapping points will not be described again.

[0062] As shown in FIG. 3, a flowchart of a method for processing visual image data in other embodiments of the present disclosure is shown. The method for processing visual image data in the embodiments of the present disclosure is applied to a cloud server and includes the following steps.

[0063] In S302, receive the image feature code stream transmitted by the terminal device. The image feature code stream is generated based on image feature information, and the image feature information is obtained by extracting features from the images in the video acquired by the terminal device. In S304, generate location information based on the image feature code stream and send it to the terminal device. In S306, receive the image code stream transmitted by the terminal device. The image code stream is generated based on the region image extracted by the terminal device based on the location information.

[0064] In some embodiments, generating location information based on the image feature code stream and sending it to the terminal device includes: decoding the image feature code stream to obtain decoded image features; inputting the decoded image features based on the residual layer of the first neural network model and outputting a first prediction result. The first prediction result includes a first prediction target category and location information.

[0065] In some embodiments, generating position information based on the image feature code stream and transmitting it to the terminal device includes querying whether the first prediction target category includes the first target category among the N sequentially set target categories, where N is a natural number, and if the first target category is not included, ending the process, and if the first target category is included, transmitting the position information to the terminal device.

[0066] Note that the cloud server needs to obtain the first prediction target category that includes the first target category. If the first target category is not included in the generated first prediction target category, it can be known that the first target category is not in the collected image, so the detection target is not in the collected image. Therefore, the terminal device needs to adjust the position or angle to re-capture the video, and the terminal device extracts the video frames to obtain a new image.

[0067] For example, in an emergency rescue scene, query whether a person's arm is injured. The preset first target category is a person. If the person is included in the first prediction target category, extract the region image, that is, the person's arm region. If the person is not included in the first prediction target category, it is necessary to re-acquire the video image.

[0068] In some embodiments, after receiving the image code stream transmitted by the terminal device, the method further includes confirming whether the detection target is included in the region image based on the image code stream.

[0069] In some embodiments, confirming whether the detection target is included in the region image based on the image code stream includes decoding the image code stream to obtain a decoded image, and Using the decoded image as input based on the second neural network model, output a second prediction result, where the second prediction result includes a second predicted target category and new position information. Determine whether the second predicted target category includes the M-th target category among the N sequentially set target categories, where M is the number of times the second predicted target category is generated plus 1, and M is less than or equal to N. If the M-th target category is included, determine whether the second predicted target category is the same as the M-th target category. If they are the same, determine whether the second predicted target category is the same as the detection target. If the second predicted target category is the same as the detection target, output a prompt to find the detection target. If the second predicted target category is not the same as the detection target, end the process. If the second predicted target category is not the same as the M-th target category, send the new position information to the terminal device, and the terminal device extracts the region image based on the new position information. If the M-th target category is not included, end the process.

[0070] Note that on the cloud server, a plurality of target categories can be set in advance. The plurality of pre-set target categories are represented by N target categories here, and the size of N is defined as required. The role of setting N target categories in advance is to gradually reduce the extracted region image, thereby finding a small target, that is, the detection target, from a large image. In machine vision, there are two definition methods for small targets. One is the relative size, for example, the horizontal width of the target size is 0.1 of the original image size, which can be regarded as a small target. The other is the definition of absolute size, that is, a target with a size less than 32*32 pixels can be regarded as a small target.

[0071] Note that the second neural network model may be a deep convolutional neural network (DCNN) model, a generative adversarial network (GAN) model, a CascadeR-CNN model, or other models capable of extracting image features. The second target category is included in the second predicted target category. The second target category is different from the second predicted target category and is the same as the detection target. By obtaining the second predicted target category in a reduced manner, the positioning of the second predicted target category, which is the same as the detection target, can be made more accurate.

[0072] Note that the cloud server can send new position information and request region images with different encoding levels.

[0073] For example, when the detection target is included in the second predicted target category but is different from the detection target and the terminal device sends new position information, the encoding quantization level of the region image to be extracted is included and sent, and different resolutions of the region image can be satisfied. For example, when sending new position information for the first time, it is included and sent that the encoding quantization level of the region image is 37. When sending new position information for the second time, it is included and sent that the encoding quantization level of the region image is 27. For example, after inputting the decoded image as input into the CascadeRCNN ResneXt 101FPN model, the second predicted target category is output. The size of the second predicted target category is smaller than the size of the first predicted target category and can be set to 5% - 50% of the area size of the first predicted target category. The detection target is included in the second predicted target category, the detection target is located within the second predicted target category, and is larger than being located within the first predicted target category, which is advantageous for accurately collecting the detection target. At the same time, the size of the second predicted target category is smaller than the size of the first predicted target category, reducing the data volume of visual image transmission and further reducing the bandwidth pressure caused by image transmission.

[0074] Hereinafter, the above embodiments will be described separately in four cases.

[0075] As a first case, the second target category is not included in the second prediction target category.

[0076] The cloud server decrypts the image code stream for the first time, obtains the decrypted image, generates the second prediction target category and new position information based on the second neural network model, the value of M is 2, and it is determined whether the second target category is included in the second prediction target category. If it is not included, it can be seen that there is no detection target.

[0077] For example, in an emergency rescue scene, when asking whether the arm of a person wearing red clothes is injured, the first target category may be a person, the second target category may be a person wearing red clothes, the region image may be an image of a person, and if there is no person wearing red clothes in the second prediction target category, it can be seen that the second target category is not included in the second prediction target category, that is, there is no person wearing red clothes, and the process ends.

[0078] As a second case, the second target category is included in the second prediction target category, the second prediction target category is the same as the second target category, and the second prediction target category is the same as the detection target.

[0079] The cloud server decrypts the image code stream for the first time, obtains the decrypted image, generates the second prediction target category and new position information based on the second neural network model, the value of M is 2, and it is determined whether the second target category is included in the second prediction target category. If it is included, it can be seen that the detection target is included in the region image. Then it is determined whether the second prediction target category is the same as the second target category. If they are the same, it is determined whether the second prediction target category is the same as the detection target. If they are the same, a prompt for finding the detection target is output. This prompt can be output in the form of an alarm such as a buzzer, a lamp, a prompt frame, etc., and is output by a method of alarm that prompts that the detection target has been found.

[0080] For example, in an emergency rescue scene, it is queried whether a person's arm is injured. The detection target is the injured arm. The first target category is a person, and the second target category is an arm, including the injured arm and the uninjured arm. The region image is an image of a person. If there is an injured arm in the second predicted target category, the second target category is included in the second predicted target category, and the second predicted target category is the same as the second target category, indicating that it is an arm. It is determined whether the second predicted target category is the same as the detection target. If the second predicted target category is the injured arm and is the same as the detection target, a prompt for finding the detection target is output to facilitate the discovery by the employee.

[0081] As a third case, the second target category is included in the second predicted target category, the second predicted target category is the same as the second target category, and the second predicted target category is different from the detection target.

[0082] The cloud server decrypts the image code stream for the first time to obtain a decrypted image, generates a second predicted target category and new position information based on the second neural network model. The value of M is 2. It is determined whether the second target category is included in the second predicted target category. If it is included, it is known that the detection target is included in the region image. It is determined whether the second predicted target category is the same as the second target category. If they are the same, it is determined whether the second predicted target category is the same as the detection target. If they are different, the process ends.

[0083] For example, in an emergency rescue scene, it is queried whether a person's arm is injured. The detection target is the injured arm. The first target category is a person, the second target category is an arm, including the injured arm and the uninjured arm. The region image is an image of a person. If there is an injured arm in the second predicted target category, the second target category is included in the second predicted target category. The second predicted target category is the same as the second target category, and it can be seen that both are arms. It is determined whether the second predicted target category is the same as the detection target. The second predicted target category is an arm, and all arms are not injured, which is different from the injured arm that is the detection target. It can be seen that the person is not injured, and the process ends.

[0084] As a fourth case, the second target category is included in the second predicted target category, and the second predicted target category is different from the second target category. The cloud server decrypts the image code stream for the first time to obtain a decrypted image, generates a second predicted target category and new position information based on the second neural network model. The value of M is 2. It is determined whether the second predicted target category is included in the second target category. If it is included, it can be seen that the detection target is included in the region image. It is determined whether the second predicted target category is the same as the second target category. If they are different, the new position information is sent to the terminal device, and the terminal device extracts the region image based on the new position information, thereby realizing the reduced extraction of the region image, that is, the ranges of several target categories are sequentially reduced, which may be a reduction in the image area. The cloud server generates new position information based on the decrypted image and re-extracts the region image with a small range.

[0085] For example, in an emergency rescue scenario, when asking whether a person's arm is injured, the preset first target category is a person, the preset second target category is a person's arm, the detection target is the injured forearm, and the second predicted target category is a person, but different from the second target category, re-extract the region image, that is, extract the person's arm. The cloud server determines new position information based on the second predicted target category, sends the position information to the terminal device, thereby re-extracts a region image with a smaller range. The terminal device re-performs region image extraction, encodes the region image, and sends the generated image code stream to the cloud server. The cloud server decodes the image code stream, generates a decoded image. If the decoded image is a person's arm and the second predicted target category obtained from the decoded image is a person's arm, and the second predicted target category is the same as the second target category, by regenerating the position information, the reduction of the extracted region image is realized, thereby finding the detection target and realizing the accurate positioning of the detection target object.

[0086] In some embodiments, the method further When the second predicted target category is the same as the detection target, send a request to obtain the original image to the terminal device.

[0087] Note that after confirming that the second predicted target category is the same as the detection target, it may be necessary to obtain the original image for confirmation, display, etc. In that case, the cloud server needs to send a request to obtain the original image to the terminal device.

[0088] When the second predicted target category includes the detection target but is different from the detection target, and the position information of the Mth target category is sent to the terminal device, the encoding quantization level of the Mth target category region image can be included and sent to meet different resolutions of the region image. For example, when sending the position information of the first target category, it is included and sent that the encoding quantization level of the first target region image is 37. When sending the position information of the second target category, it is included and sent that the encoding quantization level of the second target region image is 27.

[0089] As shown in FIG. 4, a signaling diagram of a method for processing visual image data in an embodiment of the present disclosure is shown. Specifically, it includes the following steps.

[0090] In S402, extract the features of the images in the acquired video, obtain image feature information, and generate an image feature code stream based on the image feature information. In S404, the terminal device transmits the image feature code stream to the cloud server. In S406, generate location information based on the image feature code stream. In S408, the cloud server transmits the location information to the terminal device. In S410, extract a region image based on the location information, and generate an image code stream based on the region image. In S412, the terminal device transmits the image code stream to the cloud server. In S414, check whether a detection target is included in the region image based on the image code stream.

[0091] For example, in the unmanned inspection of the power grid, the preset first target category is a suspension clip, the preset second target category is the lack of a closing pin of the suspension clip, and the detection target is the lack of a closing pin of the suspension clip.

[0092] The terminal device (i.e., the drone or the edge node) extracts frames from the collected video, writes the extracted images into an image queue, extracts features of the images in the image queue based on the CascadeRCNN ResneXt 101FPN model, the feature extraction layer is the stem layer of resneXt101, encodes the extracted image feature information, generates an image feature code stream, the feature encoding method is entropy encoding, and the feature map may be combined with an existing encoder (such as VTM12.0, AI encoding, etc.) as an image.

[0093] The cloud server receives the image feature code stream sent by the terminal device, decodes the image feature code stream to obtain decoded image features, uses the decoded image features as the input to the network layer after the stem layer of the CascadeRCNN ResneXt 101FPN model, outputs the first predicted target category and location information. If the first predicted target category does not contain a suspension clip, the cloud server directly proceeds to process the next image feature code stream until the generated first predicted target category contains a suspension clip. If the first predicted target category contains a suspension clip and is different from the first target category, the cloud server determines the location information of the preset first target category from the first predicted target category, and the cloud server sends the location information to the terminal device. The location information is the location of the detected suspension clip. The terminal device extracts the region image of the suspension clip based on the location information, encodes the region image to generate an image code stream, and sends it to the cloud server. Since multiple suspension clips may be detected, multiple region images can exist.

[0094] After receiving the image code stream, the cloud server decodes it to obtain multiple decoded images. The decoded images are region images of the suspension clip. The decoded images are used as the input to the CascadeRCNN ResneXt 101FPN model, and the second predicted target category and new location information are output. There may be multiple output second predicted target categories. The cloud server determines whether it is the same as the detection target based on the second predicted target category. If it is the same, it is found that a suspension clip with a missing closing pin needs to be inspected. If it is necessary to confirm or display the information of the original image, the cloud server interactively obtains the original image from the terminal device.

[0095] Also, during the unmanned inspection of the power grid, the vibration damper, pole tower, umbrella skirt, and suspension wire clip are inspected simultaneously. The inspected vibration damper, pole tower, umbrella skirt, and suspension wire clip are set as the first preset target category, and the possible defects occurring in each of the vibration damper, pole tower, umbrella skirt, and suspension wire are jointly set as the second target category, enabling more efficient inspection.

[0096] In this application, the cloud server and the terminal device interact with each other to determine the detection target of the small target, identify the accurate position, and impose no bandwidth pressure.

[0097] For example, in the scene of searching for a lost item, the detection target is a marked ring. This marked ring is worn on a person's hand. The first target category is a person, the second target category is a human hand, the third target category is a ring. The terminal device (i.e., a drone or an edge node) extracts frames from the collected video, writes the extracted images into an image queue, extracts features from the images in the image queue based on the CascadeRCNN ResneXt 101FPN model. The feature extraction layer is the stem layer of resneXt101, encodes the extracted image feature information to generate an image feature code stream. The feature encoding method is entropy encoding, synthesizes the feature maps into one image and inputs it into an existing encoder (such as VTM12.0, AI encoding, etc.).

[0098] The cloud server receives the image feature code stream transmitted by the terminal device, decodes the image feature code stream to obtain the decoded image features, uses the decoded image features as the input to the network layer after the stem layer of the CascadeRCNN ResneXt 101FPN model, outputs the first predicted target category and position information, the first predicted target category is a person, which is different from the detection target, transmits new position information to the terminal device, the terminal device re-extracts the region image based on the new position information, encodes the region image to generate an image code stream, the cloud server receives the image code stream transmitted by the terminal device, decodes the image code stream to generate a decoded image, uses the decoded image as the input to the CascadeRCNN ResneXt 101FPN model, outputs a new second predicted target category and new position information, the new second predicted target category is a human hand, and when the sharpness of the human hand is high, the cloud server can extract the ring worn on the hand by the human hand and determine whether the ring is the same as the detection target.

[0099] In addition, the cloud server can determine whether mutual interaction is required between the cloud server and the terminal device based on the height of the resolution of the output second predicted target category. When the resolution is high, the cloud server can perform the extraction of the detection target, and this extraction is a reduction-type extraction. When the resolution is low, the cloud server transmits new position information to the terminal device, and the terminal device extracts the region image including the detection target.

[0100] In the embodiments of the present disclosure, based on the same inventive concept, a processing device for machine vision image data is also provided as described in the following embodiments. Since the principle of problem-solving in this system embodiment is similar to that of the above method embodiment, the implementation of this system embodiment can refer to the implementation of the above method embodiment, and the overlapping points will not be described again.

[0101] FIG. 5 is a schematic diagram of a processing apparatus for visual image data according to an embodiment of the present disclosure, including a feature extraction and encoding module 51, a terminal reception module 52, an image extraction module 53, and a processing module 54.

[0102] The feature extraction and encoding module 51 is configured to extract features of an image in the acquired video, obtain image feature information, generate an image feature code stream based on the image feature information, and transmit it to a cloud server. The terminal reception module 52 is configured to receive the position information transmitted by the cloud server, where the position information is generated by the cloud server based on the image feature code stream. The image extraction module 53 is configured to extract a region image based on the position information. The processing module 54 is configured to generate an image code stream based on the region image and transmit it to the cloud server.

[0103] It should be noted that the above-mentioned feature extraction and encoding module 51, terminal reception module 52, image extraction module 53, and processing module 54 correspond to S202 - S208 in the method embodiment and are realized in the corresponding steps. The examples and application scenarios are the same, but it is not limited to the content disclosed in the above method embodiment. It should be noted that the above modules, as part of the device, can be executed by a computer system such as a set of computer-executable instructions.

[0104] In some embodiments of the present disclosure, the feature extraction module 51 extracts features of the image based on some layers of the first neural network model, generates image feature information. The image feature information is feature-encoded to generate an image feature code stream.

[0105] In some embodiments of the present disclosure, the cloud server decodes the image feature code stream and obtains the decoded image features. The cloud server decrypts the image feature code stream, obtains the decrypted image features, uses the decrypted image features as input based on the residual layer of the first neural network model, and outputs a first prediction result, where the first prediction result includes a first predicted target category and location information.

[0106] In some embodiments of the present disclosure, the feature extraction and encoding module 51 extracts frames from the collected video, generates images, writes the images into an image queue, and is configured to extract features from the images written into the image queue.

[0107] In some embodiments of the present disclosure, the terminal receiving module 52 is further configured to receive new location information, where the new location information is included in a second prediction result, and the second prediction result includes a preset second predicted target category. The second prediction result is generated by the cloud server based on the decrypted image and the second neural network model. The image extraction module 53 is configured to extract a region image based on the new location information, generate an image code stream based on the region image, and transmit it to the cloud server.

[0108] In some embodiments of the present disclosure, the terminal receiving module 52 is further configured to receive a request for obtaining an original image transmitted by the cloud server, and the original image includes a detection target.

[0109] In other embodiments of the present disclosure, based on the same inventive concept, a processing device for machine vision image data is provided as described in the following embodiments. Since the problem-solving principle of this system embodiment is similar to that of the above method embodiment, the implementation of this system embodiment can refer to the implementation of the above method embodiment, and the overlapping points will not be described again.

[0110] FIG. 6 is a schematic diagram of a visual image data processing apparatus according to another embodiment of the present disclosure. As shown in 6, the apparatus includes a cloud receiving module 61 and a position generating module 62. Here, The cloud receiving module 61 is configured to receive an image feature code stream transmitted by a terminal device, where the image feature code stream is generated based on image feature information, and the image feature information is obtained by extracting features of an image in a video acquired by the terminal device. The position generating module 62 is configured to generate position information based on the image feature code stream and transmit it to the terminal device. The cloud receiving module 61 is further configured to receive an image code stream transmitted by the terminal device, where the image code stream is generated based on a region image extracted by the terminal device based on the position information.

[0111] It should be noted that in this specification, the above-mentioned cloud receiving module 61 and position information generating module 62 are the same as the examples and application scenarios corresponding to S302 to S306 in the method embodiment implemented by the above-mentioned modules in the corresponding steps, but it should be noted that they are not limited to those disclosed in the above-mentioned method embodiment. The above modules as part of the apparatus may be executed in a computer system such as a set of computer-executable instructions.

[0112] In some embodiments of the present disclosure, the cloud receiving module 61 includes a sub-receiving module and a decoding module (not shown). Here, The sub-receiving module is configured to receive an image feature code stream transmitted by a terminal device. The decoding module decodes the image feature code stream to obtain decoded image features. Based on the decoded image features as input to the residual layer of the first neural network model, a first prediction result is output, and the first prediction result includes a first predicted target category and position information.

[0113] In some embodiments of the present disclosure, the position generation module 62 queries whether the first target category among the N target categories sequentially set in the first prediction target category is included, where N is a natural number, if the first target category is not included, the process ends, if the first target category is included, it is configured to send the position information to the terminal device.

[0114] In some embodiments of the present disclosure, the position generation module 62 is further configured to confirm whether the detection target is included in the region image based on the image code stream.

[0115] In some embodiments of the present disclosure, the position generation module 62 further decodes the image code stream to obtain a decoded image, uses the decoded image as an input based on a second neural network model to output a second prediction result, where the second prediction result includes a second prediction target category and new position information, determines whether the Mth target category among the N target categories sequentially set in the second prediction target category is included, where M is the number of times the second prediction target category is generated plus 1, and M is less than or equal to N, if the Mth target category is included, determines whether the second prediction target category is the same as the Mth target category, if the Mth target category is not included, the process is configured to end.

[0116] In some embodiments of the present disclosure, the position generation module 62 further determines whether the second prediction target category is the same as the Mth target category, if they are the same, determines whether the second prediction target category is the same as the detection target, if the second prediction target category is the same as the detection target, outputs a prompt to find the detection target, If the second predicted target category is not the same as the detection target, the process ends. If the second predicted target category is not the same as the Mth target category, new position information is transmitted to the terminal device, and the terminal device is configured to extract a region image based on the new position information.

[0117] In some embodiments of the present disclosure, the position generation module 62 is further configured to send a request for acquiring the original image to the terminal device when the second predicted target category is the same as the detection target.

[0118] Those skilled in the art can understand that various aspects of the present disclosure can be realized as a system, a method, or a program product. Therefore, various aspects of the present disclosure can be specifically realized as a complete hardware embodiment, a complete software embodiment (including firmware, microcode, etc.), or an embodiment in which aspects of hardware and software are combined, which can be generally referred to herein as a "circuit", a "module", or a "system".

[0119] Hereinafter, with reference to FIG. 7, the electronic device 700 according to this embodiment of the present disclosure will be described. The electronic device 700 shown in FIG. 7 is merely an example and does not impose any limitation on the functions and usage scope of the embodiments of the present disclosure.

[0120] As shown in FIG. 7, the electronic device 700 is represented as a general-purpose computing device. The components of the electronic device 700 may include, but are not limited to, the at least one processing unit 710, the at least one storage unit 720, and a bus 730 that connects different system components including the storage unit 720 and the processing unit 710.

[0121] Here, the memory unit stores program code executable by the processing unit 710 to execute steps according to various embodiments of the present disclosure described in the "Example of Method" section of this specification. For example, the processing unit 710 can execute the steps of the above-described method embodiments. Extract features from images in the collected video to obtain image feature information, generate an image feature code stream based on the image feature information, and transmit it to the cloud server, receive the location information transmitted from the cloud server, where the location information is generated by the cloud server based on the image feature code stream, extract a region image based on the location information, generate an image code stream based on the region image, and transmit it to the cloud server.

[0122] The processing unit 710 can execute the following steps of the above-described method embodiments, receive the image feature code stream transmitted by the terminal device, the image feature code stream is generated based on the image feature information, and the image feature information is obtained by extracting features from images in the video acquired by the terminal device, generate location information based on the image feature code stream, and transmit it to the terminal device, receive the image code stream transmitted by the terminal device, and the image code stream is generated by the terminal device extracting a region image based on the location information and based on the region image.

[0123] The memory unit 720 can include a readable medium in the form of a volatile memory unit such as a random access memory unit (RAM) 7201 and / or a cache 7202, and can further include a read-only memory unit (ROM) 7203.

[0124] The memory unit 720 can also include a program / utility 7204 having a set (at least one) of program modules 7205 including an operating system, one or more applications, other program modules, and program data, but is not limited thereto, where implementations in a network environment may be included in each or some combinations of these examples.

[0125] The bus 730 can represent one or more of several classes of bus structures including a memory cell bus or memory cell controller, a peripheral bus, a graphical acceleration port, a processing unit, or a local bus using any of a plurality of bus structures.

[0126] The electronic device 700 can also communicate with one or more external devices 740 such as a keyboard, a pointing device, a Bluetooth device, etc., and can communicate with one or more devices that enable a user to interact with the electronic device 700, and / or can communicate with any device (such as a router, a modem, etc.) that enables the electronic device 700 to communicate with one or more other computing devices. This communication can be performed via the input / output (I / O) interface 750. Also, the electronic device 700 can communicate with a public network such as a local area network (LAN), a wide area network (WAN), and / or the Internet via the network adapter 760. As shown, the network adapter 760 communicates with other modules of the electronic device 700 via the bus 730. Although not shown, it should be understood that other hardware and / or software modules including, but not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems can be used in combination with the electronic device 700.

[0127] From the description of the above embodiments, those skilled in the art can easily understand that the exemplary embodiments described in this specification can be implemented by software, or can be implemented by combining the necessary hardware with software. Therefore, the technical aspects according to the embodiments of the present disclosure may be embodied in the form of a software product stored in a non-volatile storage medium (which may be a CD-ROM, a USB disk, a mobile hard disk, etc.) containing a plurality of instructions for executing the method according to the embodiments of the present disclosure.

[0128] In an exemplary embodiment of the present disclosure, a computer-readable storage medium, which may be a readable signal medium or a readable storage medium, is also provided. A program product capable of implementing the above method of the present disclosure is stored therein. In some possible embodiments, various aspects of the present disclosure may also be realized in the form of a program product for causing a terminal device or a cloud server to execute the steps according to various embodiments of the present disclosure described in the "Example of Method" section of this specification when executed on the terminal device or the cloud server.

[0129] More specific examples of the computer-readable storage medium in the present disclosure include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0130] In the present disclosure, a computer-readable storage medium can include a data signal that propagates within a baseband carrying readable program code or as part of a carrier wave. The data signal propagated in this way can take various forms including, but not limited to, electromagnetic signals, optical signals, or any suitable combination of those described above. A readable signal medium may be any readable medium other than a computer-readable storage medium that can transmit, propagate, or send a program used by or in combination with an instruction execution system, apparatus, or device.

[0131] In some embodiments, the program code included in the computer-readable storage medium can be transmitted via any suitable medium including, but not limited to, wireless, wired, optical cable, RF, etc., or any suitable combination of those described above.

[0132] In a specific implementation, the program code for performing the operations of the present disclosure can be described in any combination of one or more programming languages including object-oriented programming languages (such as Java, C++), and also includes conventional process programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, executed as an independent package, executed partially on the user computing device and partially on a remote computing device, or executed entirely on a remote computing device or server. When related to a remote computing device, the remote computing device can be connected to the user computing device via any type of network including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, connected via the Internet using an Internet service provider).

[0133] In the above detailed description, some modules or units of the device for executing operations have been mentioned, but such division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of the above two or more modules or units can be embodied in one module or unit. Conversely, the features and functions of the above one module or unit can be further divided and embodied in a plurality of modules or units.

[0134] Also, in the accompanying drawings, each step of the method in the present disclosure is described in a specific order, but this does not require or imply that these steps must be executed in that specific order, or that all steps must be executed to achieve the desired result. Additionally or alternatively, some steps can be omitted, a plurality of steps can be merged into one step execution, or one step can be decomposed into a plurality of step executions.

[0135] From the description of the above embodiments, those skilled in the art can easily understand that the exemplary embodiments described in this specification may be implemented by software, or may be implemented by combining the necessary hardware and software. Therefore, the technical aspects according to the embodiments of the present disclosure may be embodied in the form of a software product stored in a non-volatile storage medium (which may be a CD-ROM, a USB disk, a mobile hard disk, etc.) including a plurality of instructions for executing the method according to the embodiments of the present disclosure.

[0136] A computer program product according to an embodiment of the present disclosure includes a computer program or computer instructions, and the computer program or computer instructions are loaded and executed by a processor to cause the computer to implement any of the above-described visual image data processing methods.

[0137] After considering the specification and practicing the invention disclosed herein, those skilled in the art will readily conceive of other embodiments of the present disclosure. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include known common general knowledge or conventional technical means in the art not disclosed in the present disclosure. The specification and embodiments are to be regarded as illustrative only, and the true scope of the present disclosure is indicated by the appended claims.

Claims

1. A method for processing visual image data applied to a terminal device, comprising: extracting features of an image in the acquired video to obtain image feature information, generating an image feature code stream based on the image feature information, and transmitting the image feature code stream to a cloud server; receiving position information transmitted by the cloud server, wherein the position information is generated by the cloud server based on the image feature code stream; extracting a region image based on the position information; generating an image code stream based on the region image and transmitting the image code stream to the cloud server. A method for processing visual image data, characterized by the above.

2. The step of extracting features of an image in the acquired video to obtain image feature information, generating an image feature code stream based on the image feature information, and transmitting the image feature code stream to a cloud server includes: extracting features of the image based on some layers of a first neural network model to generate image feature information; feature encoding the image feature information to generate an image feature code stream. The method for processing visual image data according to claim 1, characterized by the above.

3. The fact that the position information is generated by the cloud server based on the image feature code stream means that: the cloud server decodes the image feature code stream to obtain decoded image features; using the decoded image features as an input based on the remaining layers of a first neural network model to output a first prediction result, wherein the first prediction result includes a first predicted target category and position information. The method for processing visual image data according to claim 1, characterized by the above.

4. After generating an image code stream based on the region image and transmitting the image code stream to the cloud server, the method further includes: receiving new position information, wherein the new position information is included in a second prediction result, the second prediction result further includes a preset second predicted target category, and the second prediction result is generated by the cloud server based on a decoded image and a second neural network model; extracting a region image based on the new position information, generating an image code stream based on the region image, and transmitting the image code stream to the cloud server. The method for processing visual image data according to claim 1, characterized by the above.

5. After generating an image code stream based on the region image and transmitting it to the cloud server, the method further includes: receiving a request for obtaining an original image transmitted by the cloud server, wherein the original image includes a detection target, and the method for processing visual image data according to claim 1.

6. A method for processing visual image data applied to a cloud server, comprising: receiving an image feature code stream transmitted by a terminal device, wherein the image feature code stream is generated based on image feature information, and the image feature information is obtained by extracting features of images in a video acquired by the terminal device; generating position information based on the image feature code stream and transmitting it to the terminal device; receiving an image code stream transmitted by the terminal device; wherein the image code stream is generated by the terminal device extracting a region image based on the position information and generating it based on the region image, and the method for processing visual image data.

7. Generating position information based on the image feature code stream and transmitting it to the terminal device includes: decoding the image feature code stream to obtain decoded image features; inputting the decoded image features based on the residual layer of the first neural network model and outputting a first prediction result; wherein the first prediction result includes a first prediction target category and position information, and the method for processing visual image data according to claim 6.

8. Generating position information based on the image feature code stream and transmitting it to the terminal device includes: querying whether the first target category among N target categories sequentially set in the first prediction target category is included, where N is a natural number; ending the process when the first target category is not included; and when the first target category is included, transmitting the position information to the terminal device, and the method for processing visual image data according to claim 7.

9. After receiving the image code stream transmitted by the terminal device, the method further includes: The method for processing visual image data according to claim 6, comprising confirming whether a detection target is included in the region image based on the image code stream.

10. Confirming whether a detection target is included in the region image based on the image code stream includes: decoding the image code stream to obtain a decoded image; using the decoded image as an input based on a second neural network model to output a second prediction result, where the second prediction result includes a second predicted target category and new position information; determining whether the second predicted target category includes the Mth target category among N sequentially set target categories, where M is one more than the number of times the second predicted target category is generated, and M is less than or equal to N; when the Mth target category is included, determining whether the second predicted target category is the same as the Mth target category; when the Mth target category is not included, ending the process. The method for processing visual image data according to claim 9 is characterized by including the above.

11. Determining whether the second predicted target category is the same as the Mth target category includes: when the second predicted target category is the same as the Mth target category, determining whether the second predicted target category is the same as the detection target; when the second predicted target category is the same as the detection target, outputting a prompt for finding the detection target; when the second predicted target category is not the same as the detection target, ending the process; when the second predicted target category is not the same as the Mth target category, sending the new position information to the terminal device so that the terminal device extracts a region image based on the new position information. The method for processing visual image data according to claim 10 is characterized by including the above.

12. The method further includes: when the second predicted target category is the same as the detection target, sending a request to the terminal device to obtain the original image. The method for processing visual image data according to claim 11 is characterized by including the above.

13. A processing device for machine vision image data applied to a terminal device, The device includes a feature extraction and encoding module, a terminal receiving module, an image extraction module, and a processing module. The feature extraction and encoding module is configured to extract features from images in the acquired video, obtain image feature information, generate an image feature code stream based on the image feature information, and transmit it to a cloud server. The terminal receiving module is configured to receive the position information transmitted by the cloud server, where the position information is generated by the cloud server based on the image feature code stream. The image extraction module is configured to extract a region image based on the position information. The processing module is configured to generate an image code stream based on the region image and transmit it to the cloud server, and is a processing device for machine vision image data.

14. A processing device for machine vision image data applied to a cloud server, The device includes a cloud receiving module and a position generation module. The cloud receiving module is configured to receive the image feature code stream transmitted by a terminal device, where the image feature code stream is generated based on image feature information, and the image feature information is obtained by extracting features from images in the video acquired by the terminal device. The position generation module is configured to generate position information based on the image feature code stream and transmit it to the terminal device. The cloud receiving module is further configured to receive the image code stream transmitted by the terminal device, where the image code stream is generated by the terminal device extracting a region image based on the position information and generating it based on the region image, and is a processing device for machine vision image data.

15. An electronic device including a processor and a memory storing executable instructions of the processor, The processor is configured to execute the processing method of visual image data according to any one of Claims 1 to 12 by executing the executable instructions, and is an electronic device.

16. A computer-readable storage medium storing a computer program which, when executed by a processor, realizes the method for processing visual image data according to any one of claims 1 to 12.

17. A computer program product comprising a computer program which, when executed by a processor, realizes the method for processing visual image data according to any one of claims 1 to 12.

Images