Method, apparatus and related device for processing visual image data

Abstract

The present disclosure provides a method, apparatus, electronic device, and medium for processing visual image data. The method includes extracting features of images in the collected video to obtain image feature information, generating an image feature code stream based on the image feature information, transmitting the image feature code stream to a cloud server, receiving position information transmitted from the cloud server, where the position information is generated by the cloud server based on the image feature code stream, extracting a region image based on the position information, generating an image code stream based on the region image, and transmitting the image code stream to the cloud server. The present disclosure increases the compression ratio of the video, reduces the pressure caused by the bandwidth, enables the terminal device and the cloud server to process images simultaneously, reduces the computing amount of the cloud server, and improves the real-time processing ability.

Description

[Technical Field]

[0001] (Cross-reference of related applications) This disclosure claims priority to a Chinese patent application bearing application number 202210774919.0, filed on July 1, 2022, entitled "Method, apparatus, electronic device and medium for processing visual image data," the entire contents of which are incorporated herein by reference.

[0002] The present disclosure relates to the field of image processing technology, and in particular to a method, apparatus, electronic device, computer-readable storage medium and computer program product for processing visual image data. [Background technology]

[0003] With the rapid development of artificial intelligence technology, scenes such as smart cities, smart transportation, and unmanned patrol inspection have begun to be applied on a large scale. During application, visual images need to be collected, and terminal devices collect visual images and then transmit the collected visual images to cloud servers. The collected visual images often need to meet the characteristics of high pixel count, which causes large bandwidth pressure for visual image transmission. In addition, the collected visual images are generally processed on cloud servers, which has poor real-time processing capabilities.

[0004] It should be noted that the information disclosed in the above background art section is used only to enhance understanding of the background of the present disclosure and may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] The present disclosure provides a method, apparatus, electronic device, medium and computer program product for processing visual image data, which overcomes at least to some extent the problems of high visual image transmission bandwidth pressure and poor real-time processing capability.

[0006] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0007] One aspect of the present disclosure provides a visual image data processing method applied to a terminal device, the method including: extracting features from an image in an acquired video, acquiring image feature information, generating an image feature code stream based on the image feature information, and sending the generated image feature code stream to a cloud server; receiving location information sent by the cloud server, where the location information is generated by the cloud server based on the image feature code stream; extracting an area image based on the location information; and generating an image code stream based on the area image and sending the generated image code stream to the cloud server.

[0008] In one embodiment of the present disclosure, extracting features from images in the captured video, obtaining image feature information, and generating and transmitting an image feature code stream based on the image feature information to a cloud server includes extracting features from the images and generating image feature information based on some layers of a first neural network model, and feature-encoding the image feature information to generate an image feature code stream.

[0009] In one embodiment of the present disclosure, the location information being generated by the cloud server based on the image feature code stream includes the cloud server decoding the image feature code stream to obtain decoded image features, and outputting a first prediction result using the decoded image features as input based on a remaining layer of a first neural network model, wherein the first prediction result includes a first predicted target category and location information.

[0010] In one embodiment of the present disclosure, after generating an image code stream based on the region image and sending it to the cloud server, the method further includes: receiving new location information, wherein the new location information is included in a second prediction result, the second prediction result further includes a second predetermined predicted target category, and the second prediction result is generated by the cloud server based on the decoded image and a second neural network model; extracting a region image based on the new location information; and generating an image code stream based on the region image and sending it to the cloud server.

[0011] In one embodiment of the present disclosure, after generating an image code stream based on the region image and sending it to the cloud server, the method further includes receiving a request for acquiring an original image sent by the cloud server, wherein the original image includes the detection target.

[0012] Another aspect of the present disclosure provides a method for processing visual image data applied to a cloud server, the method including: receiving an image feature code stream transmitted by a terminal device, the image feature code stream being generated based on image feature information, the image feature information being feature extracted from an image in a video captured by the terminal device; generating location information based on the image feature code stream and transmitting it to the terminal device; and receiving the image code stream transmitted by the terminal device, the image code stream being generated based on the area image extracted by the terminal device based on the location information.

[0013] In one embodiment of the present disclosure, generating location information based on the image feature code stream and transmitting it to the terminal device includes: decoding the image feature code stream to obtain decoded image features; and outputting a first prediction result using the decoded image features as input based on a residual layer of a first neural network model, wherein the first prediction result includes a first predicted target category and location information.

[0014] In one embodiment of the present disclosure, generating location information based on the image feature code stream and transmitting it to the terminal device includes inquiring whether the first predicted target category includes a first target category among N target categories set sequentially, where N is a natural number, and if the first target category is not included, terminating the processing; and if the first target category is included, transmitting the location information to the terminal device.

[0015] In one embodiment of the present disclosure, after receiving the image code stream transmitted by the terminal device, the method further includes: determining whether the region image contains a detection object based on the image code stream.

[0016] In one embodiment of the present disclosure, confirming whether the region image includes a detection target based on the image code stream includes: decoding the image code stream to obtain a decoded image; using the decoded image as an input based on a second neural network model to output a second prediction result, wherein the second prediction result includes a second predicted target category and new position information; determining whether the second predicted target category includes an Mth target category among N target categories sequentially set, where M is the number of times the second predicted target category is generated plus 1, and M is less than or equal to N; if the Mth target category is included, determining whether the second predicted target category is the same as the Mth target category; and if the Mth target category is not included, terminating the process.

[0017] In one embodiment of the present disclosure, determining whether the second predicted target category is the same as the M target categories comprises: If the second predicted target category is the same as the M target category, determining whether the second predicted target category is the same as the detected target; If the second predicted target category is the same as the detected object, outputting a suggestion to find the detected object; If the second predicted target category is not the same as the detected target, terminating the process. If the second predicted target category is not the same as the Mth target category, transmitting the new location information to the terminal device so that the terminal device extracts a region image based on the new location information.

[0018] In one embodiment of the present disclosure, the method further includes sending a request to the terminal device to obtain an original image if the second predicted target category is the same as the detected target.

[0019] Another aspect of the present disclosure provides a machine vision image data processing device applied to a terminal device, the device including: a feature extraction encoding module, a terminal receiving module, an image extraction module, and a processing module, wherein the feature extraction encoding module is configured to extract features from an image in a captured video, obtain image feature information, and generate an image feature code stream based on the image feature information and send it to a cloud server; the terminal receiving module is configured to receive location information sent by the cloud server, where the location information is generated by the cloud server based on the image feature code stream; the image extraction module is configured to extract a region image based on the location information; and the processing module is configured to generate an image code stream based on the region image and send it to the cloud server.

[0020] Another aspect of the present disclosure provides a processing device for machine vision image data applied to a cloud server, the device including: a cloud receiving module; and a position generation module. The cloud receiving module is configured to receive an image feature code stream transmitted by a terminal device, the image feature code stream being generated based on image feature information, and the image feature information being feature extracted from an image in a video captured by the terminal device. The position generation module is configured to generate position information based on the image feature code stream and transmit it to the terminal device. The cloud receiving module is further configured to receive the image code stream transmitted by the terminal device, the image code stream being generated based on the area image extracted by the terminal device based on the position information.

[0021] Another aspect of the present disclosure provides an electronic device including a processor and a memory for storing processor executable instructions, the processor configured to perform the above-described method for processing visual image data by executing the executable instructions.

[0022] Another aspect of the present disclosure provides a computer-readable storage medium storing a computer program which, when executed by a processor, implements the above-described method for processing visual image data.

[0023] Another aspect of the present disclosure provides a computer program product including a computer program or computer instructions, which when loaded and executed by a processor causes the computer to implement any one of the above methods for processing visual image data.

[0024] In the visual image data processing method, apparatus, electronic device, medium, and computer program product according to the embodiments of the present disclosure, a terminal device extracts features from images in a collected video, obtains image feature information, and generates an image feature code stream based on the image feature information; a cloud server generates location information based on the image feature code stream; the terminal device extracts area images based on the location information, generates an image feature code stream based on the area images, and transmits the extracted image code stream to the cloud server. The image feature extraction by the terminal device and the generation of the image feature code stream based on the image feature information improves the video compression rate and reduces bandwidth pressure; the terminal device and the cloud server process the images simultaneously, reducing the computational load of the cloud server and enhancing the real-time processing capability.

[0025] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. [Brief explanation of the drawings]

[0026] The drawings herein are incorporated into the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the drawings in the following description are merely some embodiments of the present disclosure, and those skilled in the art can obtain other drawings based on these drawings without paying creative labor. [Figure 1] FIG. 1 is a schematic diagram of an application system architecture according to an embodiment of the present disclosure. [Figure 2] FIG. 1 is a flowchart illustrating a method for processing visual image data according to an embodiment of the present disclosure. [Figure 3] FIG. 10 is a flowchart illustrating a method for processing visual image data according to another embodiment of the present disclosure. [Figure 4] 1 shows a signaling diagram of a method for processing visual image data in an embodiment of the present disclosure; [Figure 5] 1 is a schematic diagram of a processing device for visual image data according to an embodiment of the present disclosure; [Figure 6] FIG. 10 is a schematic diagram of a processing device for visual image data according to another embodiment of the present disclosure. [Figure 7] FIG. 1 shows a configuration block diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0027] Next, exemplary embodiments will be described in more detail with reference to the drawings. However, exemplary embodiments may be embodied in various forms and should not be construed as being limited to the examples set forth herein. Rather, these embodiments are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0028] Furthermore, the drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings indicate the same or similar parts, and redundant description will be omitted. Some of the block diagrams shown in the drawings are functional entities that do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor and / or microcontroller devices.

[0029] Specific embodiments of the present disclosure will be described in detail below with reference to the drawings.

[0030] FIG. 1 shows a schematic diagram of an exemplary application system architecture to which the visual image data processing method according to the embodiment of the present disclosure can be applied.

[0031] As shown in FIG. 1, the system architecture includes a terminal device including a video collection device 101 for collecting video images and an image preprocessing device 102, and a cloud server 103. The video collection device 101 may be a drone, a camera, a mobile phone, or other type of device with video or image capture capabilities. The video collection device 101 is used to collect videos or images. If the video collection device 101 has image preprocessing capabilities, it is connected to a cloud server 103 via a network to preprocess the collected videos or images and transfer them to the cloud server 103. The cloud server 103 is used to process the visuals or images after processing by the video collection device 101. If the video collection device 101 does not have image preprocessing capabilities, the video collection device 101 is connected to the cloud server 103 via an image preprocessing device 102 for preprocessing the collected videos or images. The image preprocessing device 102 may be a server that provides various services, such as a background management server that supports the video collection device 101.

[0032] 1 are merely approximate, and there can be any number of video collection devices 101, image pre-processing devices 102, and cloud servers 103 according to actual needs, and the embodiments of the present disclosure are not limited thereto.

[0033] The network for connecting the image preprocessing device 102 and the cloud server 103, and the network for connecting the video collection device 101 and the cloud server 103 may be a wired network or a wireless network.

[0034] In some embodiments, the wireless or wired networks described above use standard communication technologies and / or protocols. The network is typically the Internet, but may be any network, including, but not limited to, any combination of a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a private network, or a virtual private network. In some embodiments, technologies and / or formats, including HyperText Mark-up Language (HTML), Extensible Markup Language (XML), etc., are used to represent data exchanged over the network. Also, all or some links may be encrypted using conventional encryption technologies, such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPsec), etc. In other embodiments, customized and / or proprietary data communication technologies may be used in place of or in addition to the above data communication technologies.

[0035] The cloud server 103 may be an independent physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides infrastructure cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0036] An embodiment of the present disclosure provides a method for processing visual image data, in which a terminal device extracts features from images in an acquired video, obtains image feature information, generates an image feature code stream based on the image feature information, and sends it to the cloud server 103, the cloud server 103 generates location information from the image feature code stream, and the terminal device extracts an area image from the location information, generates an image code stream based on the area image, and sends it to the cloud server 103. This method can be applied to power inspection, base station inspection, disaster relief, field scientific testing, farm and ranch monitoring, and drone filming scenes, and performs data processing on the collected videos and images to meet people's filming purposes.

[0037] In some embodiments, the visual image data processing method in the embodiments of the present disclosure can be performed in the terminal device and the cloud server 103 shown in FIG.

[0038] As shown in FIG. 2, a flowchart of a visual image data processing method in an embodiment of the present disclosure is shown, which is applied to a terminal device and includes the following steps:

[0039] In S202, features of the images in the acquired video are extracted to obtain image feature information, and an image feature code stream is generated from the image feature information and transmitted to the cloud server.

[0040] The terminal device can collect video or images, and if the collected video is video, extract frames from the collected video and write the extracted images into an image queue. The terminal device can extract features of the image through a neural network model, obtain image feature information, and encode the image feature information to generate an image feature code stream. When the image is transmitted, only the encoded image feature information is transmitted, thereby reducing the amount of transmission data of the visual image and reducing the bandwidth pressure caused by the visual image transmission.

[0041] In S204, the location information sent by the cloud server is received, where the location information is generated by the cloud server based on the image feature code stream.

[0042] The position information is the position information of the region image to be extracted, the position information is a set of several coordinate points, and the position of the region image can be determined based on the multiple coordinate points; the position information is also a combination of several coordinate points and image edge information, and the position of the region image can be determined based on the multiple coordinate points and image edge information; the image edge information may be the width of the image or the length of the image, and the position of the region image can be determined by using the combination of the coordinate points and the width of the image or the combination of the coordinate points and the length of the image.

[0043] In S206, a region image is extracted based on the position information.

[0044] After receiving the location information sent from the cloud server, the terminal device extracts an area image based on the location information, the size of the area image is smaller than the size of the image, which can be 5% to 50% of the image area size, the detection object is contained in the area image, the detection object is located in the image and is larger than the area image, which is advantageous for accurately positioning and collecting the detection object, and the detection object is the object image that needs to be extracted, and at the same time, the size of the area image is smaller than the size of the image, which can reduce the amount of transmission data of the visual image, and further reduce the bandwidth pressure caused by image transmission.

[0045] In S208, an image code stream is generated based on the region image and sent to the cloud server.

[0046] The terminal device encodes the area image to generate an image code stream, and the cloud server decodes the image code stream and determines whether the decoded area image is the same as the detection object. Through constant interaction between the terminal device and the cloud server, the required results can be obtained, such as whether the vibration-proof hammer is broken or not in an unmanned power grid inspection project.

[0047] In the related art, machine vision tasks for small targets often require multiple target area definitions and cascaded machine vision tasks, and image collection in these scenes often requires high-resolution features, resulting in significant bandwidth pressure for video transmission. Furthermore, video transmission requires cloud servers to perform computational processing, and cloud servers have limited computing power. When the amount of data to be processed is large, the real-time processing capability is weakened, affecting work efficiency.

[0048] The present disclosure extracts image features at the terminal device and generates an image code stream based on the image feature information, thereby increasing the video compression rate and reducing bandwidth pressure; the terminal device and the cloud server simultaneously process images collaboratively, reducing the computational load of the cloud server and improving real-time processing capabilities.

[0049] In some embodiments, extracting features from images in the captured video, obtaining image feature information, generating an image feature code stream based on the image feature information, and transmitting to the cloud server includes: extracting features from an image based on a portion of layers of a first neural network model to generate image feature information; and feature-encoding the image feature information to generate an image feature code stream.

[0050] The first neural network model may be a deep convolutional neural network (DCNN) model, a generative adversarial network (GAN) model, a CascadeR-CNN model, or other models capable of image feature extraction.

[0051] For example, the terminal device extracts features from an image through the stem layer of resneXt101 of the CascadeRCNN ResneXt 101 FPN model, generates image feature information, encodes the image feature information through a feature encoding method, and generates an image feature code stream, where the feature encoding method can be one of all-intra (AI) encoding, clustering quantization, and entropy encoding.

[0052] In some embodiments, the location information is generated by a cloud server based on the image feature code stream, The cloud server decodes the image feature code stream to obtain decoded image features; The method includes taking the decoded image features as input based on a residual layer of the first neural network model and outputting a first prediction result, wherein the first prediction result includes a first predicted target category and location information.

[0053] For the first predicted target category, it is pre-configured in the cloud server, and the location information is used to determine the location of the area image, facilitating the terminal device to extract the area image based on the location information.

[0054] For example, the decoded image features are input to a network layer after the stem layer of the Cascade RCNN ResneXt 101FPN model, and a first prediction result is output. The first prediction result includes a first predicted target category and location information, and the location information can be expressed as a coordinate set.

[0055] In some embodiments, extracting features from images in the captured video, obtaining image feature information, generating an image feature code stream based on the image feature information, and transmitting to the cloud server includes: extracting frames from the collected video to generate images; writing the image to the image queue; and characterizing the images written to the image queue.

[0056] For image generation, the terminal device collects video, frames the video, generates a plurality of images, writes the images to an image queue, sequentially stacks the images located in the image queue, and the terminal device performs feature extraction on the images stacked.

[0057] In some embodiments, after generating and transmitting the image code stream based on the region image to the cloud server, the method further comprises: receiving new location information, the new location information being included in a second prediction result, the second prediction result including a second predetermined predicted target category, the second prediction result being generated by the cloud server based on the decoded image and the second neural network model; Extracting a region image based on the new location information, generating an image code stream based on the region image, and transmitting the image code stream to a cloud server.

[0058] In addition, the cloud server decodes the image code stream, obtains the decoded image, and outputs the decoded image as an input to a second neural network model to produce a second prediction result, where the second prediction result includes new location information.

[0059] In some embodiments, an image code stream is generated based on the region image and transmitted to the cloud server, and the method further comprises: The method includes receiving a request to acquire an original image sent by the cloud server, and determining that the original image includes the detection target.

[0060] If the second predicted target category is the same as the detected target, the cloud server sends a request to obtain the original image to the terminal device to check or display the original image.

[0061] Based on the same inventive concept, the embodiments of the present disclosure also provide a method for processing visual image data, as described in the following embodiments. Since the problem-solving principle of the present method embodiment is similar to that of the above-mentioned method embodiment, the implementation of the present method embodiment can refer to the implementation of the above-mentioned method embodiment, and the overlapping points will no longer be described.

[0062] As shown in FIG. 3 , a flowchart of a method for processing visual image data in another embodiment of the present disclosure is shown, where the method for processing visual image data in the embodiment of the present disclosure is applied to a cloud server, and includes the following steps:

[0063] In S302, an image feature code stream transmitted by a terminal device is received, the image feature code stream being generated based on image feature information, and the image feature information is extracted by extracting features of an image in a video captured by the terminal device; In S304, position information is generated based on the image feature code stream and sent to the terminal device; In S306, an image code stream sent by the terminal device is received, and the image code stream is generated based on the region image extracted by the terminal device based on the position information.

[0064] In some embodiments, generating and transmitting location information based on the image feature code stream to the terminal device comprises: Decoding the image feature codestream to obtain decoded image features; Based on a residual layer of a first neural network model, the decoded image features are input, and a first prediction result is output, where the first prediction result includes a first predicted target category and location information.

[0065] In some embodiments, generating and transmitting location information based on the image feature code stream to the terminal device comprises: Inquiring whether the first predicted target category includes the first target category among N target categories set sequentially, where N is a natural number; If the first target category is not included, terminating the process; If the first target category is included, transmitting the location information to the terminal device.

[0066] In addition, the cloud server needs to obtain a first predicted target category that includes the first target category. If the generated first predicted target category does not include the first target category, it can be determined that the first target category is not in the collected image. Since the detection target is not in the collected image, the terminal device needs to adjust the position or angle and re-image the video, and the terminal device extracts frames from the video to obtain new images.

[0067] For example, in an emergency rescue scene, it inquires whether a person's arm is injured, and the preset first target category is a person. If the first predicted target category contains a person, it performs area image extraction, i.e., a person's arm area. If the first predicted target category does not contain a person, it is necessary to re-acquire the video image.

[0068] In some embodiments, after receiving the image codestream transmitted by the terminal device, the method further comprises: This includes determining whether the target is contained within the region image based on the image codestream.

[0069] In some embodiments, determining whether the region image includes a target based on the image codestream comprises: decoding the image codestream to obtain a decoded image; Using the decoded image as an input based on a second neural network model, outputting a second prediction result, the second prediction result including a second predicted target category and new position information; Determine whether the second predicted target category includes an Mth target category among the N target categories set sequentially, where M is the number of times the second predicted target category is generated plus 1, and M is equal to or less than N; If an Mth target category is included, determining whether the second predicted target category is the same as the Mth target category; If so, determining whether the second predicted target category is the same as the detected target; If the second predicted target category is the same as the detected object, outputting a suggestion to find the detected object; If the second predicted target category is not the same as the detected target, terminating the process. If the second predicted target category is not the same as the Mth target category, sending new location information to the terminal device, and the terminal device extracts a region image based on the new location information; and if the Mth target category is not included, terminating the process.

[0070] Note that multiple target categories can be preset on the cloud server. These preset target categories are represented here as N target categories, where N is defined as needed. The purpose of presetting the N target categories is to gradually shrink the extracted area image, thereby finding small targets, i.e., detection targets, from large images. In machine vision, there are two ways to define small targets: one is relative size (for example, a target whose width is 0.1 of the original image size can be considered a small target); the other is absolute size (i.e., a target whose size is less than 32*32 pixels can be considered a small target).

[0071] In addition, the second neural network model may be a deep convolutional neural network (DCNN) model, a generative adversarial network (GAN) model, a CascadeR-CNN model, or other models capable of extracting image features, and the second target category is included in the second predicted target category, and the second target category is different from the second predicted target category. By obtaining the second predicted target category that is the same as the detection target in a reduced manner, the positioning of the second predicted target category that is the same as the detection target can be more accurate.

[0072] Note that the cloud server can request area images with different coding levels along with sending new location information.

[0073] For example, when the second predicted target category includes the detection target but is different from the detection target, new location information is transmitted to the terminal device, including the encoding quantization level of the region image to be extracted, so as to accommodate different resolutions of the region image. For example, when transmitting new location information the first time, the encoding quantization level of the region image is transmitted, including that it is 37, and when transmitting new location information the second time, the encoding quantization level of the region image is transmitted, including that it is 27. For example, the decoded image is input to the Cascade RCNN ResneXt 101FPN model, and then a second predicted target category is output. The size of the second predicted target category is smaller than that of the first predicted target category, and can be 5% to 50% of the area size of the first predicted target category. The detected object is included in the second predicted target category, and the detected object is located in the second predicted target category and is larger than that located in the first predicted target category. This is advantageous for accurately collecting the detected object, and at the same time, the size of the second predicted target category is smaller than that of the first predicted target category, which reduces the data volume of visual image transmission and further reduces the bandwidth pressure caused by image transmission.

[0074] The above embodiment will be explained below in four cases.

[0075] In the first case, the second predicted target category does not include the second target category.

[0076] The cloud server first decodes the image code stream to obtain the decoded image, and generates a second predicted target category and new location information based on the second neural network model, where M is 2. The cloud server determines whether the second predicted target category includes the second target category. If not, it is determined that there is no target to be detected.

[0077] For example, in an emergency rescue scene, if an inquiry is made as to whether the arm of a person wearing red clothes is injured, the first target category may be a person, the second target category may be a person wearing red clothes, and the region image may be an image of a person; if there is no person wearing red clothes in the second predicted target category, it is found that the second predicted target category does not include the second target category, that is, there is no person wearing red clothes, and the processing is terminated.

[0078] In the second case, the second predicted target category includes the second target category, the second predicted target category is the same as the second target category, and the second predicted target category is the same as the detection target.

[0079] The cloud server first decodes the image code stream to obtain the decoded image, and generates a second predicted target category and new location information based on the second neural network model, where M is 2. The cloud server determines whether the second predicted target category contains the second target category. If so, it is determined that the area image contains the detected object. The cloud server determines whether the second predicted target category is the same as the second target category. If so, it determines whether the second predicted target category is the same as the detected object. If so, it outputs a notification that the detected object has been found. The notification can be output in the form of an alarm such as a buzzer, a lamp, or a notification frame, indicating that the detected object has been found.

[0080] For example, in an emergency rescue scene, inquiring whether a person's arm is injured, the detection target is an injured arm, the first target category is a person, the second target category is an arm, including an injured arm and an uninjured arm, the region image is an image of a person, if there is an injured arm in the second predicted target category, the second predicted target category includes a second target category, the second predicted target category is the same as the second target category, it is found to be an arm, it is determined whether the second predicted target category is the same as the detection target, the second predicted target category is an injured arm, and the detection target and the injured arm are the same, a suggestion to find the detection target is output, making it easier for employees to find it.

[0081] As a third case, the second predicted target category includes the second target category, the second predicted target category is the same as the second target category, and the second predicted target category is different from the detection target.

[0082] The cloud server first decodes the image code stream to obtain a decoded image, and generates a second predicted target category and new location information based on a second neural network model, where the value of M is 2. The cloud server determines whether the second target category is included in the second predicted target category, and if so, it is determined that the region image contains the detected target. The cloud server determines whether the second predicted target category is the same as the second target category, and if so, determines whether the second predicted target category is the same as the detected target. If not, the process ends.

[0083] For example, in an emergency rescue scene, inquiring whether a person's arm is injured, the detection target is an injured arm, the first target category is a person, the second target category is an arm, including injured and uninjured arms, the region image is an image of a person, if there is an injured arm in the second predicted target category, the second predicted target category includes the second target category, the second predicted target category is the same as the second target category, it is found that both are arms, it is determined whether the second predicted target category is the same as the detection target, the second predicted target category is an arm, all arms are not injured, it is found that unlike the injured arm which is the detection target, people will not be injured, and the processing is terminated.

[0084] As a fourth case, the second predicted target category includes the second target category, and the second predicted target category is different from the second target category; The cloud server first decodes the image code stream to obtain a decoded image, and generates a second predicted target category and new location information based on a second neural network model, where M is 2. The cloud server determines whether the second predicted target category is included in the second target category. If so, it is determined that the detected object is included in the area image. The cloud server determines whether the second predicted target category is the same as the second target category. If not, it sends new location information to the terminal device, and the terminal device extracts the area image based on the new location information, thereby realizing reduced extraction of the area image. That is, the ranges of some target categories are sequentially reduced, which may result in a reduction in the image area. The cloud server generates new location information based on the decoded image and re-extracts the area image with a smaller range.

[0085] For example, in an emergency rescue scene, when inquiring whether a person's arm is injured, the preset first target category is a person, the preset second target category is a human arm, the detected object is an injured forearm, and the second predicted target category is a person but different from the second target category, then re-extracting the region image, i.e., extracting the human arm, the cloud server determines new location information based on the second predicted target category and sends the location information to the terminal device, thereby re-extracting a region image with a smaller range, the terminal device re-extracts the region image, encodes the region image, and sends the generated image code stream to the cloud server, the cloud server decodes the image code stream to generate a decoded image, the decoded image is a human arm, the second predicted target category obtained from the decoded image is a human arm, and the second predicted target category is the same as the second target category, and re-generating the location information reduces the extracted region image, thereby finding the detected object and accurately positioning the detected target.

[0086] In some embodiments, the method further comprises: If the second predicted target category is the same as the detected target, a request to acquire the original image is sent to the terminal device.

[0087] After confirming that the second predicted target category is the same as the detected target, it may be necessary to acquire the original image for confirmation or display, in which case the cloud server must send a request to acquire the original image to the terminal device.

[0088] The second predicted target category includes the detection target, but is different from the detection target. When transmitting location information of the Mth target category to the terminal device, the encoding quantization level of the Mth target category region image can be included and transmitted to meet different resolutions of the region image. For example, when transmitting the first target category location information, the encoding quantization level of the first target region image can be included and transmitted as 37, and when transmitting the second target category location information, the encoding quantization level of the second target region image can be included and transmitted as 27.

[0089] As shown in Figure 4, a signaling diagram of a method for processing visual image data in an embodiment of the present disclosure is shown. Specifically, it includes the following steps:

[0090] In S402, extract features of images in the acquired video to obtain image feature information, and generate an image feature code stream based on the image feature information; In S404, the terminal device sends the image feature code stream to the cloud server; In S406, position information is generated based on the image feature code stream; In S408, the cloud server sends the location information to the terminal device; In S410, extracting a region image based on the position information, and generating an image codestream based on the region image; In S412, the terminal device sends the image code stream to the cloud server; In S414, it is determined whether the detection target is included in the region image based on the image code stream.

[0091] For example, in an unmanned inspection of a power grid, the first preset target category is catenary clip, the second preset target category is missing closing pin of the catenary clip, and the detection target is missing closing pin of the catenary clip.

[0092] The terminal device (i.e., drone or edge node) extracts frames from the collected video, writes the extracted images into an image queue, and performs feature extraction on the images in the image queue based on the Cascade RCNN ResneXt 101 FPN model. The feature extraction layer is the stem layer of ResneXt 101, encodes the extracted image feature information, and generates an image feature code stream. The feature encoding method is entropy encoding, and the feature map can be combined into an existing encoder (e.g., VTM12.0, AI encoding, etc.) as an image.

[0093] The cloud server receives the image feature code stream sent by the terminal device, decodes the image feature code stream to obtain decoded image features, and uses the decoded image features as input for the network layer after the stem layer of the Cascade RCNN ResneXt 101FPN model, outputting a first predicted target category and location information. If the first predicted target category does not include a catenary clip, the cloud server proceeds directly to processing the next image feature code stream until the generated first predicted target category includes a catenary clip. If the first predicted target category includes a catenary clip and the first predicted target category is different from the first target category, the cloud server determines location information for a predetermined first target category from the first predicted target category. The cloud server sends the location information to the terminal device, which location information is the location of the detected catenary clip. The terminal device extracts a region image of the catenary clip based on the location information. The terminal device image-encodes the region image, generates an image code stream, and sends it to the cloud server. Since multiple catenary clips may be detected, there may be multiple region images.

[0094] After receiving the image code stream, the cloud server decodes it to obtain multiple decoded images, which are catenary clip area images. The decoded images are input to the Cascade RCNN ResneXt 101FPN model, which outputs a second predicted target category and new position information. There may be multiple output second predicted target categories. The cloud server determines whether the second predicted target category is the same as the detected target based on the second predicted target category. If the second predicted target category is the same, it performs a patrol inspection to find the catenary clip with the missing closing pin. If the original image information needs to be confirmed or displayed, the cloud server interactively obtains the original image from the terminal device.

[0095] In addition, during unmanned inspections of power grids, the anti-vibration weights, pillar towers, umbrella skirts, and suspension wire clips are inspected simultaneously, and the inspected anti-vibration weights, pillar towers, umbrella skirts, and suspension wire clips are simultaneously set as a pre-set first target category, and defects that may occur in each of the anti-vibration weights, pillar towers, umbrella skirts, and suspension wires are jointly set as a second target category, thereby achieving more efficient inspections.

[0096] The present application allows the cloud server and the terminal device to interact with each other to determine the target to be detected for small targets, pinpoint their precise location, and not cause pressure on bandwidth.

[0097] For example, in a scene of searching for lost property, the detection target is a marked ring, which is worn by a person's hand. The first target category is a person, the second target category is a hand, and the third target category is a ring. The terminal device (i.e., a drone or edge node) extracts frames from the collected video, writes the extracted images into an image queue, and performs feature extraction on the images in the image queue based on the Cascade RCNN ResneXt 101 FPN model. The feature extraction layer is the stem layer of ResneXt 101, which encodes the extracted image feature information and generates an image feature code stream. The feature encoding method is entropy encoding, and the feature map is synthesized into a single image and input into an existing encoder (e.g., VTM12.0, AI encoding, etc.).

[0098] The cloud server receives the image feature code stream sent by the terminal device, decodes the image feature code stream to obtain decoded image features, and uses the decoded image features as inputs of a network layer after the stem layer of the CascadeRCNN ResneXt 101FPN model, outputting a first predicted target category and location information. The first predicted target category is a person, and the first predicted target category is different from the detected target. The cloud server sends new location information to the terminal device, and the terminal device re-extracts a region image based on the new location information, encodes the region image, and generates an image code stream. The cloud server receives the image code stream sent by the terminal device, decodes the image code stream to generate a decoded image, and uses the decoded image as inputs of the CascadeRCNN ResneXt 101FPN model, outputting a new second predicted target category and new location information. The new second predicted target category is a human hand. If the definition of the human hand is high, the cloud server can extract a ring worn by the human hand and determine whether the ring is the same as the detected target.

[0099] In addition, the cloud server can determine whether mutual interaction is required between the cloud server and the terminal device based on the resolution of the output second predicted target category. If the resolution is high, the cloud server can perform extraction of the detection target, which is a reduced extraction. If the resolution is low, the cloud server sends new location information to the terminal device, and the terminal device extracts an area image including the detection target.

[0100] Based on the same inventive concept, the embodiments of the present disclosure also provide a machine vision image data processing device, as described in the following embodiments. Since the problem-solving principle of this system embodiment is similar to that of the above-mentioned method embodiment, the implementation of this system embodiment can refer to the implementation of the above-mentioned method embodiment, and the overlapping points will no longer be described.

[0101] FIG. 5 is a schematic diagram of a visual image data processing device in an embodiment of the present disclosure, including a feature extraction encoding module 51, a terminal receiving module 52, an image extraction module 53, and a processing module 54.

[0102] The feature extraction encoding module 51 is configured to extract features of images in the captured video, obtain image feature information, generate an image feature code stream based on the image feature information, and send the generated image feature code stream to the cloud server; the terminal receiving module 52 is configured to receive location information sent by the cloud server, the location information being generated by the cloud server based on the image feature code stream; The image extraction module 53 is configured to extract a region image based on the position information; A processing module 54 is configured to generate and transmit an image code stream based on the region image to the cloud server.

[0103] It should be noted that the above-mentioned feature extraction encoding module 51, terminal receiving module 52, image extraction module 53 and processing module 54 correspond to S202 to S208 in the method embodiment, and the examples and application scenarios realized by the corresponding steps are the same as those disclosed in the above-mentioned method embodiment. It should be noted that the above-mentioned modules can be executed by a computer system, such as a set of computer-executable instructions, as part of an apparatus.

[0104] In some embodiments of the present disclosure, the feature extraction module 51 extracts features from the image based on some layers of a first neural network model to generate image feature information; The image feature information is feature-coded to generate an image feature code stream.

[0105] In some embodiments of the present disclosure, the cloud server decodes the image feature code stream to obtain the decoded image features; The cloud server decodes the image feature code stream to obtain the decoded image features; Based on a residual layer of a first neural network model, the decoded image features are input, and a first prediction result is output, where the first prediction result includes a first predicted target category and location information.

[0106] In some embodiments of the present disclosure, the feature extraction and encoding module 51 extracts frames from the collected video to generate images, Write the image to the image queue, It is configured to feature images written to the image queue.

[0107] In some embodiments of the present disclosure, the terminal receiving module 52 is further configured to receive new location information, the new location information being included in a second prediction result, the second prediction result including a second predetermined predicted target category, the second prediction result being generated by the cloud server based on the decoded image and the second neural network model; The image extraction module 53 is configured to extract a region image based on the new location information, generate an image code stream based on the region image, and send it to the cloud server.

[0108] In some embodiments of the present disclosure, the terminal receiving module 52 is further configured to receive an acquisition request for an original image sent by the cloud server, where the original image includes the detection target.

[0109] In another embodiment of the present disclosure, based on the same inventive concept, a machine vision image data processing apparatus is provided, as described in the following embodiments. Since the problem-solving principle of this system embodiment is similar to that of the above-mentioned method embodiment, the implementation of this system embodiment can refer to the implementation of the above-mentioned method embodiment, and the overlapping points will no longer be described.

[0110] FIG. 6 is a schematic diagram of a visual image data processing device in another embodiment of the present disclosure. As shown in FIG. 6, the device includes a cloud receiving module 61 and a position generating module 62, where: The cloud receiving module 61 is configured to receive an image feature code stream sent by a terminal device, the image feature code stream being generated based on image feature information, and the image feature information is feature extraction of an image in a video captured by the terminal device; a location generation module 62 configured to generate location information based on the image feature code stream and transmit the location information to the terminal device; The cloud receiving module 61 is further configured to receive an image code stream sent by the terminal device, the image code stream being generated by the terminal device based on the extracted area image based on the location information.

[0111] It should be noted that in this specification, the above-mentioned cloud receiving module 61 and location information generating module 62 are the same as the examples and application scenarios corresponding to S302 to S306 in the method embodiments in which the above-mentioned modules are implemented in the corresponding steps, but are not limited to those disclosed in the above-mentioned method embodiments. The above-mentioned modules as part of an apparatus may be executed in a computer system such as a set of computer-executable instructions.

[0112] In some embodiments of the present disclosure, the cloud receiving module 61 includes a sub-receiving module and a decoding module (not shown), where: The sub-receiving module is configured to receive an image feature code stream transmitted by a terminal device; a decoding module for decoding the image feature code stream to obtain a decoded image feature; Based on a residual layer of a first neural network model, the decoded image features are input, and a first prediction result is output, where the first prediction result includes a first predicted target category and location information.

[0113] In some embodiments of the present disclosure, the position generation module 62 queries whether the first predicted target category includes a first target category among N sequentially set target categories, where N is a natural number; If the first target category is not included, terminating the process; If the first target category is included, the location information is configured to be transmitted to the terminal device.

[0114] In some embodiments of the present disclosure, the position generation module 62 is further configured to determine whether the region image includes a target based on the image codestream.

[0115] In some embodiments of the present disclosure, the position generation module 62 further decodes the image codestream to obtain a decoded image; Using the decoded image as an input based on a second neural network model, output a second prediction result, the second prediction result including a second predicted target category and new position information; determining whether the second predicted target category includes an Mth target category among the N target categories sequentially set, where M is the number of times the second predicted target category is generated plus 1, and M is equal to or less than N; If an Mth target category is included, determining whether the second predicted target category is the same as the Mth target category; If the Mth target category is not included, the process is configured to end.

[0116] In some embodiments of the present disclosure, the position generation module 62 further determines whether the second predicted target category is the same as the Mth target category; If so, determining whether the second predicted target category is the same as the detected target; If the second predicted target category is the same as the detected object, outputting a suggestion to find the detected object; If the second predicted target category is not the same as the detected target, then terminating the process; If the second predicted target category is not the same as the Mth target category, new location information is sent to the terminal device, and the terminal device is configured to extract the region image based on the new location information.

[0117] In some embodiments of the present disclosure, the position generation module 62 is further configured to send a request to the terminal device to obtain an original image if the second predicted target category is the same as the detected target.

[0118] Those skilled in the art will appreciate that various aspects of the present disclosure may be embodied as a system, method, or program product. Accordingly, various aspects of the present disclosure may be embodied as an entirely hardware embodiment, an entirely software embodiment (including firmware, microcode, etc.), or an embodiment combining hardware and software aspects, which may be collectively referred to herein as a "circuit," "module," or "system."

[0119] An electronic device 700 according to this embodiment of the present disclosure will now be described with reference to Fig. 7. The electronic device 700 shown in Fig. 7 is merely an example and does not impose any limitations on the functionality and scope of use of the embodiment of the present disclosure.

[0120] 7, electronic device 700 is depicted as a general-purpose computing device. Components of electronic device 700 may include, but are not limited to, at least one processing unit 710, at least one storage unit 720, and a bus 730 connecting different system components including storage unit 720 and processing unit 710.

[0121] Here, the storage unit stores program code executable by the processing unit 710 to perform steps according to various embodiments of the present disclosure described in the "Example Method" section above of this specification. For example, the processing unit 710 can perform steps of the above-mentioned method embodiments, such as: extracting features from images in the collected video to obtain image feature information; generating an image feature code stream based on the image feature information and sending it to a cloud server; receiving location information sent from the cloud server, where the location information is generated by the cloud server based on the image feature code stream; extracting area images based on the location information; generating an image code stream based on the area images and sending it to the cloud server.

[0122] The processing unit 710 can perform the following steps of the above-mentioned method embodiment: receive an image feature code stream sent by a terminal device, the image feature code stream being generated based on image feature information, the image feature information being feature extracted from an image in a video captured by the terminal device, generate location information based on the image feature code stream and send it to the terminal device, receive an image code stream sent by the terminal device, the image code stream being extracted by the terminal device based on the location information and generated based on the area image.

[0123] The storage unit 720 may include readable media in the form of volatile storage units, such as a random access storage unit (RAM) 7201 and / or a cache 7202 , and may also include a read-only storage unit (ROM) 7203 .

[0124] The storage unit 720 may also include, but is not limited to, a program / utility 7204 having a set (at least one) of program modules 7205, including an operating system, one or more applications, other program modules, and program data, each of which, or some combination thereof, may include the implementation of a network environment.

[0125] Bus 730 may represent one or more of several classes of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphical acceleration port, a processing unit, or a local bus using any of a number of bus structures.

[0126] The electronic device 700 may also communicate with one or more external devices 740, such as a keyboard, pointing device, Bluetooth device, one or more devices that allow a user to interact with the electronic device 700, and / or any device (e.g., router, modem, etc.) that allows the electronic device 700 to communicate with one or more other computing devices. This communication may occur via an input / output (I / O) interface 750. The electronic device 700 may also communicate with a public network, such as a local area network (LAN), a wide area network (WAN), and / or the Internet, via a network adapter 760. As shown, the network adapter 760 communicates with other modules of the electronic device 700 via a bus 730. Although not shown, it should be understood that other hardware and / or software modules may be used in combination with the electronic device 700, including, but not limited to, microcode, device drives, redundant processing units, external disk drive arrays, RAID systems, tape drives, data backup storage systems, etc.

[0127] From the above description of the embodiments, it is easy for those skilled in the art to understand that the exemplary embodiments described herein can be realized by software, or by combining necessary hardware with software. Therefore, the technical aspects according to the embodiments of the present disclosure may be embodied in the form of a software product that may be stored in a non-volatile storage medium (which may be a CD-ROM, a USB disk, a mobile hard disk, etc.) and includes multiple instructions for executing the methods according to the embodiments of the present disclosure.

[0128] In an exemplary embodiment of the present disclosure, a computer-readable storage medium, which may be a readable signal medium or a readable storage medium, is also provided, having stored thereon a program product capable of implementing the above-described methods of the present disclosure. In some possible embodiments, various aspects of the present disclosure may also be implemented in the form of a program product that, when executed on a terminal device or a cloud server, causes the terminal device or the cloud server to perform steps according to various embodiments of the present disclosure described in the "Example Method" section above of this specification.

[0129] More specific examples of computer-readable storage media in this disclosure include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0130] In this disclosure, a computer-readable storage medium may include a propagating data signal, either in baseband or as part of a carrier wave, that carries readable program code. Such a propagating data signal may take various forms, including, but not limited to, an electromagnetic signal, an optical signal, or any suitable combination of the foregoing. A readable signal medium may be any readable medium other than a readable storage medium that can transmit, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device.

[0131] In some embodiments, the program code contained in the computer-readable storage medium may be transmitted over any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.

[0132] In specific implementations, program code for performing the operations of the present disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​(e.g., Java, C++), and conventional process programming languages ​​such as "C" or similar programming languages. The program code can execute entirely on the user computing device, partially on the user device, as a separate package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. When referring to a remote computing device, the remote computing device can be connected to the user computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., connected via the Internet using an Internet Service Provider).

[0133] Although the above detailed description refers to several modules or units of an apparatus for performing operations, such division is not mandatory. In fact, according to embodiments of the present disclosure, the features and functions of two or more of the above-described modules or units may be embodied in one module or unit. Conversely, the features and functions of one of the above-described modules or units may be further divided and embodied in multiple modules or units.

[0134] Additionally, although the accompanying figures may depict steps of the methods in this disclosure in a particular order, this does not require or imply that the steps must be performed in that particular order, or that all steps must be performed to achieve a desired result. Additionally or alternatively, some steps may be omitted, multiple steps may be merged into a single step execution, or a single step may be broken down into multiple step executions.

[0135] As will be readily understood by those skilled in the art from the above description of the embodiments, the exemplary embodiments described herein may be realized by software or by combining necessary hardware and software. Therefore, the technical aspects of the embodiments of the present disclosure may be embodied in the form of a software product that may be stored in a single non-volatile storage medium (which may be a CD-ROM, a USB disk, a mobile hard disk, etc.) and that includes multiple instructions for executing the methods of the embodiments of the present disclosure.

[0136] A computer program product according to an embodiment of the present disclosure includes a computer program or computer instructions, which are loaded and executed by a processor to cause the computer to implement any of the above-described methods for processing visual image data.

[0137] Other embodiments of the present disclosure will be readily apparent to those skilled in the art after considering the specification and practicing the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of the present disclosure in accordance with the general principles of the present disclosure, including common knowledge or customary technical means known in the art but not disclosed in the present disclosure. The specification and embodiments are considered to be exemplary only, with the true scope of the present disclosure being indicated by the appended claims.

Claims

1. A method for processing visual image data applied to a terminal device, comprising: extracting features from images in the video captured by the terminal device, obtaining image feature information, generating an image feature code stream based on the image feature information, and sending the generated image feature code stream to a cloud server; The terminal device receives location information sent by the cloud server, and the location information is generated by the cloud server based on the image feature code stream; The terminal device extracts a region image based on the location information; The terminal device generates an image code stream based on the region image and transmits the image code stream to the cloud server; After generating an image code stream based on the region image and sending it to the cloud server, the visual image data processing method further comprises: the terminal device receives new location information, the new location information is included in a second prediction result, the second prediction result further includes a second predetermined predicted target category, and the second prediction result is generated by a cloud server based on a decoded image of the image code stream and a second neural network model; extracting a region image again based on the new position information, and generating an image code stream based on the re-extracted region image and transmitting the image code stream to the cloud server.

2. extracting features from images in a video captured by the terminal device, acquiring image feature information, generating an image feature code stream based on the image feature information, and transmitting the generated image feature code stream to a cloud server; The terminal device extracts features from the image based on a portion of layers of a first neural network model to generate image feature information; 2. The method of claim 1, further comprising the step of: said terminal device feature coding said image feature information to generate an image feature code stream.

3. After the terminal device generates an image code stream based on the area image and sends it to the cloud server, the visual image data processing method further comprises: The method for processing visual image data according to claim 1 , further comprising: receiving, by the terminal device, a request for acquiring an original image sent by the cloud server; and the original image including a detection target.

4. A method for processing visual image data applied to a cloud server, comprising: The cloud server receives an image feature code stream transmitted by a terminal device, the image feature code stream is generated based on image feature information, and the image feature information is extracted features of an image in a video captured by the terminal device; The cloud server generates location information based on the image feature code stream and sends it to the terminal device; The cloud server receives an image code stream transmitted by the terminal device, and the image code stream is generated based on an area image extracted by the terminal device based on the location information; and the cloud server transmitting new location information to the terminal device, causing the terminal device to extract a region image again based on the new location information, wherein the new location information is included in a second prediction result, the second prediction result further includes a second predetermined predicted target category, and the second prediction result is generated by the cloud server based on a decoded image of the image code stream and a second neural network model.

10. A method for processing visual image data, comprising:

5. The cloud server generating location information based on the image feature code stream and transmitting the location information to the terminal device, the cloud server decodes the image feature code stream to obtain decoded image features; and the cloud server outputs a first prediction result using the decoded image features as an input based on remaining layers other than some layers of a first neural network model; 5. The method of claim 4, wherein the first prediction result includes a first predicted target category and location information.

6. The cloud server generating location information based on the image feature code stream and transmitting the location information to the terminal device, The cloud server determines whether the first predicted target category includes a first target category among N target categories sequentially set, where N is a natural number; If the first target category is not included, the cloud server terminates processing; and The method of claim 5, further comprising: if the first target category is included, the cloud server transmitting the location information to a terminal device.

7. After the cloud server receives the image code stream sent by the terminal device, the visual image data processing method further comprises: The method of claim 4 , further comprising: determining whether the region image includes a target based on the image code stream.

8. The cloud server confirming whether the region image includes a detection target based on the image code stream comprises: the cloud server decrypting the image codestream to obtain the decrypted image; the cloud server inputs the decoded image based on the second neural network model and outputs the second prediction result, wherein the second prediction result includes the second predicted target category and the new location information; The cloud server determines whether the second predicted target category includes an Mth target category among N target categories sequentially set, where M is the number of times the second predicted target category is generated plus 1, and M is equal to or less than N; If the M target categories are included, the cloud server determines whether the second predicted target category is the same as the M target category; and if the M target categories are not included, the cloud server terminates the process.

9. The cloud server determining whether the second predicted target category is the same as the M target category comprises: If the second predicted target category is the same as the Mth target category, the cloud server determines whether the second predicted target category is the same as the detected target; If the second predicted target category is the same as the detected object, the cloud server outputs a suggestion to find the detected object; 9. The method for processing visual image data according to claim 8, further comprising: if the second predicted target category is not the same as the detected target, the cloud server terminates the processing; and if the second predicted target category is not the same as the Mth target category, the cloud server sends the new location information to the terminal device so that the terminal device extracts an area image again based on the new location information.

10. The method for processing visual image data further comprises: The method for processing visual image data according to claim 9, further comprising: if the second predicted target category is the same as the detected target, the cloud server sends a request to the terminal device to obtain an original image.

11. 1. An electronic device including a processor and a memory storing executable instructions for said processor, An electronic device, characterized in that the processor is configured to perform the method for processing visual image data according to any one of claims 1 to 3 by executing the executable instructions.

12. An electronic device including a processor and a memory storing executable instructions for said processor, An electronic device, characterized in that the processor is configured to perform the method for processing visual image data according to any one of claims 4 to 10 by executing the executable instructions.

13. A computer-readable storage medium storing a computer program which, when executed by a processor, implements the method for processing visual image data according to any one of claims 1 to 3.

14. A computer-readable storage medium storing a computer program which, when executed by a processor, implements the method for processing visual image data described in any one of claims 4 to 10.

Images