Image data segmentation and transmission

Abstract

A computing device is provided, including a logic subsystem having one or more processors and a memory storing instructions executable by the logic subsystem. The instructions are executed to obtain one or more source images, segment the one or more source images to generate a plurality of fragments, determine a priority order for the plurality of fragments, and transmit the plurality of fragments to a remote computing device in the priority order. The plurality of fragments are spatial components generated by spatial decomposition of the one or more source images and / or frequency components generated by frequency decomposition of the one or more source images. The remote computing device can receive the components in the priority order and perform certain algorithms on the individual components without waiting for the entire image to be uploaded.

Description

Background Art

[0001] Transferring images or large data files from edge devices to cloud servers is a significant bottleneck for many cloud-based applications. For example, a drone with a camera may collect a large amount of data in a short period of time, and transferring such a large amount of data to the cloud may take several days in some cases. Transferring data to cloud servers can be challenging, especially for edge devices located in remote environments where network capacity and bandwidth are constrained. Summary of the invention

[0002] A computing device is provided, including a logic subsystem having one or more processors and a memory storing instructions executable by the logic subsystem. The instructions are executable to obtain one or more source images, segment the one or more source images to generate a plurality of fragments, determine a priority order for the plurality of fragments, and transmit the plurality of fragments to a remote computing device in the priority order. The plurality of fragments are spatial components generated by spatial decomposition of the one or more source images and / or frequency components generated by frequency decomposition of the one or more source images.

[0003] This summary is provided to introduce in simplified form a series of concepts that will be further described in the detailed description below. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to implementations that solve any or all of the disadvantages noted in any part of this disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] Figure 1 shows that it can be implemented Figure 1 Schematic diagram of an example computing environment for edge devices.

[0005] Figure 2 is a diagram illustrating segmentation of a source image into different spatial components by a spatial component algorithm according to an example.

[0006] Figure 3A and Figure 3B is an illustration of the labeling of visual features of a simulated image as part of the training of an algorithm used to generate spatial components.

[0007] Figure 4 is a diagram illustrating segmenting a source image into different frequency components by a frequency component algorithm according to an example.

[0008] Figure 5 is a graphical representation of different levels of frequency decomposition performed by a frequency component algorithm according to one example.

[0009] Fig. 6Ais a flow chart illustrating an embodiment of a first method of segmenting a captured source image into spatial components for transmission.

[0010] Figure 6B is an illustration of a flowchart detailing steps in a first method of transmitting spatial components in priority order according to a first example.

[0011] Figure 6C is an illustration of a flowchart detailing steps in a first method of transmitting spatial components in priority order according to a second example.

[0012] Fig. 7A is a flow chart illustrating an embodiment of a second method of splitting a captured source image into frequency components and spatial components for transmission.

[0013] Figure 7B is a flow chart illustrating an embodiment of a third method of splitting a captured source image into frequency components and spatial components for transmission.

[0014] Figure 7C is an illustration of a flowchart detailing steps in a second method and a third method of determining a priority order for transmitting frequency components and spatial components according to one example.

[0015] Fig.7D is an illustration of one example implementation of an algorithm that may be performed on different frequency components of a source image when the different frequency components are received in a priority order by a remote computing device.

[0016] Fig. 8A is a flow chart illustrating an embodiment of a fourth method of segmenting a captured source image into only frequency components for transmission.

[0017] Figure 8B is a flow chart illustrating an embodiment of a fifth method of splitting a captured source image into frequency components and spatial components for transmission.

[0018] Fig. 9 is a flow chart illustrating an embodiment of a sixth method of splitting audio data into frequency components for transmission.

[0019] Fig.10 A computing system according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0020] Figure 1An example use environment 100 is schematically shown, in which an edge device 102 captures images of the real world using one or more cameras 104, which are embodied as image capture devices in this example use environment 100. The camera 104 can be installed on a fixed device or a mobile device such as a vehicle, a drone, a satellite, and a machine. A remote computing device 106, which can be configured as a cloud server, stitches the images acquired by the one or more cameras 104 together. The image capture device 102 can include a component that communicatively couples the device to one or more other computing devices 106, which can be a cloud server. For example, the image capture device 102 can be communicatively coupled to (multiple) other computing devices 106 via a network 108. In some examples, the network 108 can take the form of a local area network (LAN), a wide area network (WAN), a wired network, a wireless network, a personal area network, or a combination thereof, and can include the Internet.

[0021] The edge device 102 is configured to divide the captured image into a plurality of segments, determine that one or more of the segments are high priority for the image processing application 134 executing on the edge device 102. The edge device 102 then determines a given priority order to transmit or upload the segments to the one or more computing devices 106 based on the determination that the one or more segments are high priority for the image processing application 134, and then transmits the segments to the one or more computing devices 106 based on the given priority order. In some embodiments, the captured image may be filtered by the edge device 102 to select a subset of the plurality of captured images to segment and transmit to the one or more computing devices 106, wherein the subset of the plurality of captured images is a target object for analysis.

[0022] The image capture device 102 includes one or more cameras 104, each camera 104 acquiring one or more images of the use environment 100. In some examples, the camera(s) 104 include one or more visible light cameras configured to capture visible light image data from the use environment 100. Example visible light cameras include RBG cameras and / or grayscale cameras. The camera(s) 104 may also include one or more depth image sensors configured to capture depth image data for the use environment 100. Example depth image sensors include infrared time-of-flight depth cameras and associated infrared illuminators, infrared structured light depth cameras and associated infrared illuminators, and stereo camera arrangements.

[0023] The image capture device 102 may be communicatively coupled to a display 110, which may be integrated with the image capture device 102 (e.g., within a shared housing) or may be a peripheral device to the image capture device 102. The image capture device 102 may also include one or more electro-acoustic transducers or speakers 112 to output audio. In one specific example where the image capture device 102 is used as a video conferencing device, the speakers 112 receive audio from the computing device(s) 106 and output the received audio so that participants in the use environment 100 can hold a video conference with one or more remote participants associated with the computing device(s) 106. In addition, the image capture device 102 may include one or more microphones 114 that receive audio data 116 from the use environment 100. Although in Figure 1 102, but in other examples, one or more of the microphone(s) 114, the camera(s) 104, and / or the speaker(s) 112 may be separate from the image capture device and communicatively coupled to the image capture device 102.

[0024] The image capture device 102 includes a segmentation application 118, which may be stored in the mass storage device 120 of the image capture device 102. The segmentation application 118 may be loaded into the memory 122 and executed by the processor 124 of the image capture device 102 to perform one or more methods and processes described in more detail below. An image and / or audio processing application 134 may also be stored in the mass storage device 120 of the image capture device 102 and is configured to process segments or components of an image generated by the segmentation application 118.

[0025] The segmentation application 118 processes one or more source images captured by the image capture device 102 and generates a plurality of segments based on the one or more source images using a spatial component algorithm 126 for generating spatial components and / or a frequency component algorithm 130 for generating frequency components. The plurality of segments are generated as spatial components generated by spatial decomposition of the one or more source images and / or frequency components generated by frequency decomposition of the one or more source images. The plurality of spatial components may identify spatial features within the source images. These spatial features are not particularly limited and may include landmarks, man-made structures, vegetation, bodies of water, the sky, and other visual features captured in the source images.

[0026] Although Figure 1A mass storage device 120 having a segmentation application 118 and an image / audio processing application 134 is depicted in an image capture device 102, and the image capture device 102 is configured to transmit image components to (multiple) computing devices 106, but it should be understood that, similar to the image capture device 102, the (multiple) computing devices 106 can also be configured with a mass storage device and a segmentation application to store image data, so that the segmentation application executed on the (multiple) computing device 106 can generate components based on the stored image data and transmit the components to the edge device 102 or other computing devices via the network 108 in a priority order.

[0027] Steering Figure 2 In a first implementation of generating spatial components, multiple spatial components can be generated based on domain knowledge. For example, the segmentation application 118 can incorporate a spatial component algorithm 126, which is configured to identify features within the source image and generate spatial components accordingly. As used herein, the spatial component of an image refers to the portion of the image that is clipped from the image. For example, the spatial component algorithm can contain logic that identifies specific features and draws a clipping boundary (i.e., a path) around the image area where the object is located. For example, the image can be modified by extracting image data within the clipping boundary and storing it as a spatial component in a separate file (or a block within an integrated file). In an agricultural embodiment, the segmentation application 118 can incorporate a spatial component algorithm 126, which is configured to identify trees, grass, and the sky, and generate spatial components corresponding to these identified features. Therefore, a spatial component corresponding to a tree, a spatial component corresponding to grass, and a spatial component corresponding to the sky are generated by the segmentation application 118. The spatial component algorithm 126 can be generated via a machine learning algorithm or a simulator, which is trained on an image of a visual feature (with a label corresponding to the visual feature). As an example, a simulator trained on visual features of trees, grass, and farmland can be configured to generate spatial components corresponding to trees, grass, and farmland. Figure 3A As shown in , a machine learning algorithm can be trained on simulated images to label parts of the simulated images as ground. Figure 3B As illustrated in , a machine learning algorithm can be further trained on the generated spatial components to identify features within the spatial components, such as plants or vegetation.

[0028] In a second implementation of generating spatial components, multiple spatial components can be generated based on manual input. The segmentation application 118 can execute the spatial component algorithm 126 to receive input from the user identifying features within the source image and generate spatial components accordingly. For example, in an agricultural embodiment, the user can input data identifying trees, grass, and sky in the source image into the segmentation application 118, and the segmentation application 118 can generate spatial components based on the data input by the user. Therefore, in this example, a spatial component corresponding to a tree, a spatial component corresponding to grass, and a spatial component corresponding to the sky can be generated by the segmentation application 118.

[0029] In an implementation of determining a priority order for transmitting components, multiple components may be generated by the segmentation application 118, and then the priority order for the components may be determined by the segmentation application 118 based on the application sensitivity algorithm 132. The application sensitivity algorithm 132 may add noise to each of the components, or degrade the quality of each of the components, and determine the effect of the noise addition or quality degradation of each of the components on the indication monitored by the application sensitivity algorithm 132. For example, the indication may be a quantitative measurement of an object captured in the source image, such as the number of fruits or insects. When the indication changes beyond a predetermined threshold or crosses a predetermined threshold, the segmentation application determines that the component is a high priority. In an example where the indication is the number of fruits, when the number of fruits measured in the spatial component corresponding to the tree drops sharply due to the degradation of the component, this change in the indication may result in an indication that crosses a predetermined threshold, thereby determining that the spatial component corresponding to the tree is a high priority. It should be understood that the application sensitivity algorithm 132 may be applied to the spatial components and the frequency components to determine the priority order in which the spatial components and the frequency components are transmitted to the remote computing device so that the highest priority or most important component is received by the remote computing device first.

[0030] Steering Figure 4 , when the segmentation application 118 processes one or more source images captured by the image capture device 102 and generates a plurality of segments based on the one or more source images, the plurality of segments may also include a plurality of frequency components, which are generated by applying the frequency component algorithm 130 to perform frequency decomposition on the source images. In other words, the image may be decomposed based on the frequency of visual characteristics. That is, blocks of the image may be transformed into the frequency domain, and the frequency coefficients may be quantized and entropy encoded. Alternatively, the frequency decomposition may be performed as a degradation of the image quality of the image, a wavelet-based decomposition, or any decomposition technique in which there is a basis for applying each increment to improve the image quality.

[0031] like Figure 5As illustrated in , in one implementation of frequency decomposition, three frequency components are generated for a source image via three different frequency decomposition techniques: high frequency decomposition, low frequency decomposition, and DC basis. It should be appreciated that there is no particular limit to the number of frequency components, and more than three frequency decomposition fragments may be generated. The frequency at which the frequency decomposition techniques are performed may be adjusted depending on the amount of available bandwidth to transmit the image components to a remote computing device via a network, such that the resolution of each frequency component depends on the amount of available bandwidth. For example, the source image may be encoded in a compression format that inherently supports frequency decomposition, such as, but not limited to, JPEG XR, JPEG 2000, or AV1. Support for frequency decomposition in these compression formats lies in the fact that these encoding mechanisms identify frequencies for their own encodings.

[0032] refer to Figure 5 , when the frequency components are transmitted by the segmentation application to the remote computing device, the frequency components corresponding to the DC basis can be transmitted first (level 3), then the frequency components corresponding to the low frequency decomposition can be transmitted (level 2), then the frequency components corresponding to the high frequency decomposition can be transmitted (level 1), and then the remote computing device can stitch or assemble the original image together after receiving all the frequency components of the source image (level 0). It should be understood that the remote computing device can perform processing tasks on the image without waiting for the original image to be reconstructed. For example, in an agricultural embodiment involving fruit trees, at level 3, fruit counting can be performed. At level 2, yield prediction can be performed. At level 1, genotyping can be performed. Therefore, tasks suitable for image quality or fidelity achieved at certain stages in the image upload process are performed.

[0033] Preferably, frequency decomposition is performed on the source image, and then spatial decomposition is performed to generate the segments, as discussed further below. For example, three frequency components may be generated by the segmentation application 118, and then three spatial components may be generated for each of the generated frequency components. However, it should be appreciated that in other embodiments, only spatial decomposition may be performed (e.g., Fig. 6A 500) or only perform frequency decomposition (e.g., Fig. 8A Alternatively, a spatial decomposition may be performed on the source image and then a frequency decomposition may be performed to generate the segments, such as Figure 8B As described in.

[0034] Fig. 6A A flowchart of a first method 500 for transmitting or uploading an image in segments to a remote computing device according to an example of the present disclosure is illustrated. In the first method 500, a source image is transmitted to a remote computing device using only spatial components. The following description of the method 500 is based on the above description and is Figure 1It should be appreciated that method 500 may also be performed in other contexts using other suitable hardware and software components.

[0035] At 502, the edge device obtains one or more source images as image data. At 504, the segmentation application of the edge device segments the one or more source images to generate spatial components by spatial decomposition of the one or more source images. At 506, the segmentation application determines a priority order for a plurality of segments to transmit the spatial components to a remote computing device. At 508, the segmentation application transmits the spatial components to the remote computing device in a priority order. At 510, the remote computing device receiving the spatial components in a priority order first receives the highest priority component. At 512, the remote computing device performs an operation on the highest priority component. This operation may be an analysis task to derive useful information from the highest priority component. At 514, when the remote computing device receives the remaining components in a priority order, the remote computing device stitches or assembles the components to complete the image transmission. For example, when the low frequency components of a tree are stitched or assembled with the high frequency components of the tree, the image quality of the stitched or assembled tree image at the remote computing device is improved to be close to the source image quality. When additional bandwidth becomes available in a network connecting the edge device to the remote computing device, lower priority components can be transmitted to the remote computing device.

[0036] Figure 6B A first example of a flowchart detailing step 508 of the first method 500 is illustrated, wherein the spatial components are transmitted to the remote computing device in the priority order determined in step 506. At 508A, the edge device may first transmit the low-quality component of the entire image. After transmitting the low-quality component of the entire image, at 508B, the edge device may transmit the high-quality, high-priority component of the image. For example, in an agricultural embodiment, the high-priority component may be a spatial component of a farmland or a tree, which may be a target for analysis by the remote computing device. After transmitting the high-quality, high-priority component of the image, at 508C, the edge device may transmit the high-quality, low-priority component of the image.

[0037] Figure 6CA second example of a flowchart detailing step 508 of the first method 500 is illustrated, wherein the spatial components are transmitted to the remote computing device in the priority order determined in step 506. At 508A, the edge device may first transmit a high-quality image of the highest priority component. In an agricultural embodiment, the highest priority component may be a spatial component of a field or a tree, which may be a target for analysis by the remote computing device. After transmitting the highest priority component, at 508B, the edge device may transmit a high-quality image of the second-highest priority component. For example, in an agricultural embodiment, the second-highest priority component may be a spatial component of grass. After transmitting the second-highest priority component of the image, at 508C, the edge device may transmit a high-quality image of the lowest priority component of the image. For example, in an agricultural embodiment, the lowest priority component may be a spatial component of the sky.

[0038] Fig. 7A A flow chart of a second method 600 for transmitting or uploading an image in segments to a remote computing device according to an example of the present disclosure is illustrated. In the second method 600, both frequency components and spatial components are used to transmit a source image to a remote computing device, wherein the frequency components are first generated and then the spatial components are generated for each of the frequency components. The following description of the method 600 is based on the above description and is Figure 1 It should be appreciated that method 600 may also be performed in other contexts using other suitable hardware and software components.

[0039] At 602, the edge device obtains one or more source images as image data. At 604, the segmentation application of the edge device segments the one or more source images to generate frequency components by frequency decomposition of the obtained images. At 606, the segmentation application segments the one or more source images to generate spatial components for each of the frequency components by spatial decomposition of the one or more source images. At 608, the segmentation application determines a priority order for a plurality of fragments to transmit the frequency components and the spatial components to a remote computing device. At 610, the segmentation application transmits the plurality of fragments (frequency components and spatial components) to the remote computing device in priority order. At 612, the remote computing device that receives the frequency components and the spatial components in priority order first receives the highest priority component. At 614, the remote computing device performs an operation on the highest priority component. This operation may be an analysis task that derives useful information from the highest priority component. At 616, when the remote computing device receives the remaining components in priority order, the remote computing device stitches or assembles the components to complete the image transmission.

[0040] Figure 7BA flow chart of a third method 700 for applying the second method 600 in an agricultural application for counting fruit on a tree according to an example of the present disclosure is illustrated. In the third method 700, both frequency components and spatial components are used to transmit a source image to a remote computing device. The following description of the method 700 is in reference to the above description and is Figure 1 It should be understood that method 700 can also be performed in other contexts using other suitable hardware and software components. For the sake of brevity, the steps on the remote computing device side are omitted.

[0041] At 702, the edge device obtains one or more source images, such as trees. At 704, the segmentation application of the edge device segments the one or more source images to generate frequency components, such as low-frequency components, high-frequency components, and DC base, by frequency decomposition of the obtained source images of the trees. At 706, the segmentation application segments the one or more source images to generate spatial components, such as sky components, tree components, and grass components, for each of the frequency components by spatial decomposition of the one or more source images. At 708, the segmentation application determines a priority order for a plurality of fragments to transmit the frequency components and the spatial components to a remote computing device. At 710, the segmentation application transmits the plurality of fragments (frequency components and spatial components) to the remote computing device in a priority order. In this example, for an application that counts fruits on a tree, the tree is considered a high priority. Therefore, the priority order is determined, for example, to be DC base tree, low frequency tree, high frequency tree, DC base grass, low frequency grass, high frequency grass, DC base sky, low frequency sky, and high frequency sky in this order. The sky components are transmitted last because they provide the least important data related to the application that counts fruits.

[0042] Figure 7C 7 also describes the application sensitivity algorithm 132 used by the segmentation application 118 of the edge device 102 .

[0043] At 706A, each component is obtained. At 706B, an operation is performed by the image processing application 134 on each component to obtain or monitor an indication returned as a result of the operation performed on the component. At 706C, the quality of each component is degraded, or noise is added to each component. At 706D, an operation is again performed by the image processing application 134 on each component to obtain or monitor an indication returned as a result of the operation performed on the component. At 706E, the segmentation application identifies a priority component having a change in indication exceeding a predetermined threshold, or a priority component having an indication crossing a predetermined threshold. At 706F, the segmentation application determines a priority order, which priority component is the first component to be transmitted to the remote computing device.

[0044] Fig.7D An example implementation of an algorithm that can be performed on different frequency components of a source image when the different frequency components are received by a remote computing device in a priority order is illustrated. In this example, four frequency components are generated by the segmentation application 118 executing the frequency component algorithm 130: an LQ base, which has a magnitude ratio of 1X; an MQ delta, which has a magnitude ratio of 10X; an HQ delta, which has a magnitude ratio of 40X; and an EQ delta, which has a magnitude ratio of 80X. The edge device transmits the LQ base frequency component, the MQ delta frequency component, the HQ delta frequency component, and the EQ delta frequency component to the remote computing device in this order.

[0045] When the remote computing device completes uploading 5% of the source image, the uploading of the LQ base frequency component may have been completed. Therefore, after the uploading of the LQ base frequency component is completed, the remote computing device may perform a fruit counting algorithm on the LQ base frequency component to implement fruit counting, because the image quality of the LQ base frequency component may be sufficient to implement an acceptable fruit counting.

[0046] Subsequently, when the remote computing device completes uploading 20% ​​of the source image, the uploading of the MQ incremental frequency components may have been completed. Therefore, after the uploading of the MQ incremental frequency components is completed, the remote computing device may execute a yield prediction algorithm on the MQ incremental frequency components to perform yield prediction, because the image quality of the MQ incremental frequency components may be sufficient to perform an acceptable yield prediction.

[0047] Subsequently, when the remote computing device completes uploading 80% of the source image, the uploading of the HQ delta frequency component may have been completed. Therefore, after the uploading of the HQ delta frequency component is completed, the genotyping algorithm may be executed on the HQ delta frequency component by the remote computing device to perform genotyping, because the image quality of the HQ delta frequency component may be sufficient to perform genotyping. Subsequently, after the uploading of the entire source image is completed, the entire source image may be archived by the remote computing device for further processing at a later time.

[0048] Fig. 8A A flowchart of a fourth method 800 for transmitting or uploading an image in segments to a remote computing device according to an example of the present disclosure is illustrated. In the fourth method 800, a source image is transmitted to a remote computing device using only frequency components. The following description of the method 800 is based on the above description and is Figure 1 It should be appreciated that method 800 may also be performed in other contexts using other suitable hardware and software components.

[0049] At 802, an edge device obtains one or more source images as image data. At 804, a segmentation application of the edge device segments the one or more source images to generate frequency components by frequency decomposing the obtained images. At 806, the segmentation application determines a priority order for multiple fragments to transmit the frequency components to a remote computing device. At 808, the segmentation application transmits multiple fragments (frequency components) to the remote computing device in priority order. At 810, the remote computing device that receives the frequency components in priority order first receives the highest priority component. At 812, the remote computing device performs an operation on the highest priority component. This operation can be an analysis task that derives useful information from the highest priority component. At 814, when the remote computing device receives the remaining components in priority order, the remote computing device stitches or assembles the components to complete the image transmission.

[0050] Figure 8B A flow chart of a fifth method 900 for transmitting or uploading an image in segments to a remote computing device according to an example of the present disclosure is illustrated. In the fifth method 900, both frequency components and spatial components are used to transmit a source image to a remote computing device. However, unlike the second method 600 in which frequency components are first generated and then spatial components are generated for each of the frequency components, in the fifth method 900, spatial components are first generated and then frequency components are generated for each of the spatial components. The following description of the method 900 is with reference to the above description and is Figure 1 It should be appreciated that method 900 may also be performed in other contexts using other suitable hardware and software components.

[0051] At 902, the edge device obtains one or more source images as image data. At 904, the segmentation application of the edge device segments the one or more source images to generate spatial components by spatial decomposition of the obtained images. At 906, the segmentation application segments the one or more source images to generate frequency components for each of the spatial components by frequency decomposition of the one or more source images. At 908, the segmentation application determines a priority order for a plurality of fragments to transmit the frequency components and the spatial components to a remote computing device. At 910, the segmentation application transmits the plurality of fragments (frequency components and spatial components) to the remote computing device in priority order. At 912, the remote computing device that receives the frequency components and the spatial components in priority order first receives the highest priority component. At 914, the remote computing device performs an operation on the highest priority component. This operation may be an analysis task that derives useful information from the highest priority component. At 916, when the remote computing device receives the remaining components in priority order, the remote computing device stitches or assembles the components to complete the image transmission.

[0052] It should be understood that segmentation applications are not limited to segmenting images. Fig. 9 , a flowchart of a sixth method 1000 for transmitting or uploading an image in segments to a remote computing device according to an example of the present disclosure is depicted. In the sixth method 1000, source audio is transmitted to the remote computing device using only frequency components. The following description of method 1000 is based on the above description and is Figure 1 It should be appreciated that method 1000 may also be performed in other contexts using other suitable hardware and software components.

[0053] At 1002, the edge device obtains source audio. Figure 1 As shown in , the source audio may be audio data 116 captured by a microphone 114. At 1004, a segmentation application of the edge device segments the source audio to generate frequency components by frequency decomposing the obtained source audio. At 1006, the segmentation application determines a priority order for a plurality of segments to transmit the frequency components to a remote computing device. At 1008, the segmentation application transmits the plurality of segments (frequency components) to the remote computing device in priority order. At 1010, a remote computing device that receives frequency components in priority order first receives the highest priority component. At 1012, the remote computing device performs an operation on the highest priority component. This operation may be an analysis task that derives useful information from the highest priority component. At 1014, when the remote computing device receives the remaining components in priority order, the remote computing device stitches or assembles the components to complete the audio transmission.

[0054] Thus, an image compression system is described in which the system automatically detects image segments that are important for analysis in a cloud server and transmits these segments with a higher priority than other less important segments. The original image data can be encoded in a progressive format, which means that because important segments of the image increase in fidelity at a faster rate than other less important segments, the image can be understood immediately on the remote computing device. This can achieve the potential advantage of exponentially improved latency in image transmission between edge devices and remote computing devices.

[0055] It should be understood that the above-described methods and computing devices may also be applied to other fields besides the agricultural field, such as oil and gas extraction, fishing, search and rescue, security systems, etc.

[0056] In some embodiments, the methods and processes described herein may be bound to a computing system of one or more computing devices. In particular, such methods and processes may be implemented as computer applications or services, application programming interfaces (APIs), libraries, and / or other computer program products.

[0057] Fig.10 A non-limiting embodiment of a computing system 1100 is schematically shown, which can implement one or more of the above-described methods and processes. The computing system 1100 is shown in simplified form. The computing system 1100 can embody Figure 1 The computing device 106 or image capture device 102. The computing system 1100 may take the form of one or more personal computers, server computers, tablet computers, home entertainment computers, network computing devices, gaming devices, mobile computing devices, mobile communication devices (e.g., smart phones), and / or other computing devices and wearable computing devices such as smart watches and head-mounted augmented reality devices.

[0058] The computing system 1100 includes a logic processor 1102, a volatile memory 1104, and a non-volatile storage device 1106. The computing system 1100 may optionally include a display subsystem 1108, an input subsystem 1110, a communication subsystem 1112, and / or other components not described herein. Fig.10 Other components shown in FIG.

[0059] Logical processor 1102 includes one or more physical devices configured to execute instructions. For example, a logical processor may be configured to execute instructions that are part of one or more applications, programs, routines, libraries, objects, components, data structures, or other logical constructs. Such instructions may be implemented to perform a task, implement a data type, transform the state of one or more components, implement a technical effect, or otherwise achieve a desired result.

[0060] The logical processor may include one or more physical processors (hardware) configured to execute software instructions. Additionally or alternatively, the logical processor may include one or more hardware logic circuits or firmware devices configured to execute hardware-implemented logic or firmware instructions. The processor of the logical processor 1102 may be single-core or multi-core, and the instructions executed thereon may be configured for sequential processing, parallel processing, and / or distributed processing. The various components of the logical processor may optionally be distributed between two or more separate devices, which may be remotely located and / or configured for coordinated processing. Various aspects of the logical processor may be virtualized and executed by a remotely accessible networked computing device configured as a cloud computing configuration. In this case, it is understandable that these virtualized aspects run on different physical logical processors of various different machines.

[0061] The non-volatile storage device 1106 includes one or more physical devices configured to store instructions that can be executed by a logical processor to implement the methods and processes described herein. When implementing such methods and processes, the state of the non-volatile storage device 1106 can be converted - for example, to store different data.

[0062] The non-volatile storage device 1106 may include a removable and / or built-in physical device. The non-volatile storage device 1106 may include an optical memory (e.g., CD, DVD, HD-DVD, Blu-ray disc, etc.), a semiconductor memory (e.g., ROM, EPROM, EEPROM, flash memory, etc.), and / or a magnetic memory (e.g., a hard disk drive, a floppy disk drive, a tape drive, MRAM, etc.) or other mass storage device technology. The non-volatile storage device 1106 may include a non-volatile, dynamic, static, read / write, read-only, sequential access, location addressable, file addressable, and / or content addressable device. It should be understood that the non-volatile storage device 1106 is configured to save instructions even when the non-volatile storage device 1106 is powered off.

[0063] The volatile memory 1104 may include physical devices that include random access memory. The volatile memory 1104 is typically utilized by the logical processor 1102 to temporarily store information during the processing of software instructions. It should be understood that when power is removed from the volatile memory 1104, the volatile memory 1104 typically does not continue to store instructions.

[0064] Aspects of the logic processor 1102, volatile memory 1104, and non-volatile storage device 1106 may be integrated together into one or more hardware logic components. Such hardware logic components may include field programmable gate arrays (FPGAs), program and application specific integrated circuits (PASIC / ASIC), program and application specific standard products (PSSP / ASSP), systems on chips (SOCs), and complex programmable logic devices (CPLDs).

[0065] The terms "module", "program", and "engine" may be used to describe aspects of the computing system 1100 that are typically implemented in software by a processor to perform a specific function using portions of volatile memory, which involves a conversion process that specifically configures the processor to perform the function. Thus, a module, program, or engine may be instantiated using portions of volatile memory 1104 via logical processor 1102 to execute instructions stored by non-volatile storage device 1106. It should be understood that different modules, programs, and / or engines may be instantiated from the same application, service, code block, object, library, routine, API, function, etc. Similarly, the same module, program, and / or engine may be instantiated by different applications, services, code blocks, objects, routines, APIs, functions, etc. The terms "module", "program", and "engine" may cover a single executable file or a group of executable files, a data file, a library, a driver, a script, a database record, etc.

[0066] When included, the display subsystem 1108 can be used to present a visual representation of the data stored by the non-volatile storage device 1106. The visual representation can take the form of a graphical user interface (GUI). Since the methods and processes described herein change the data stored by the non-volatile storage device, and thus convert the state of the non-volatile storage device, the state of the display subsystem 1108 can also be converted to visually represent the changes in the underlying data. The display subsystem 1108 can include one or more display devices utilizing almost any type of technology. Such a display device can be combined with a logical processor 1102, a volatile memory 1104, and / or a non-volatile storage device 1106 in a shared shell, or such a display device can be a peripheral display device.

[0067] When included, the input subsystem 1110 may include or interface with one or more user input devices, such as a keyboard, mouse, touch screen, or game controller. In some embodiments, the input subsystem may include or interface with selected natural user input (NUI) components. Such components may be integrated or peripheral, and the transformation and / or processing of input actions may be handled on-board or off-board. Example NUI components may include microphones for speech and / or voice recognition; infrared, color, stereo, and / or depth cameras for machine vision and / or gesture recognition; head trackers, eye trackers, accelerometers, and / or gyroscopes for motion detection and / or intent recognition; and electric field sensing components for assessing brain activity; and / or any other suitable sensors.

[0068] When included, the communication subsystem 1112 can be configured to communicatively couple the various computing devices described herein to each other and to communicatively couple to other devices. The communication subsystem 1112 can include wired and / or wireless communication devices compatible with one or more different communication protocols. As a non-limiting example, the communication subsystem can be configured to communicate via a wireless telephone network, or a wired or wireless local area network or a wide area network (such as Bluetooth and HDMI connected via Wi-Fi). In some embodiments, the communication subsystem can allow the computing system 1100 to send messages to other devices and / or receive messages from other devices via a network such as the Internet.

[0069] It should be understood that "and / or" as used herein refers to a logical disjunction operation, so A and / or B has the following truth table.

[0070] A B A and / or B T T T T F T F T T F F F

[0071] The following paragraphs provide additional support for the claims of the present application. One aspect provides a method, comprising: obtaining one or more source images as image data at an edge device; segmenting the one or more source images to generate a plurality of fragments; determining a priority order for the plurality of fragments; and transmitting the plurality of fragments to a remote computing device in a priority order, the plurality of fragments being generated as spatial components generated by spatial decomposition of the one or more source images and / or frequency components or more source images generated by frequency decomposition of the one or more source images. In this regard, additionally or alternatively, the frequency decomposition may be a degradation of image quality. In this regard, additionally or alternatively, the frequency decomposition may be a decomposition of the frequency of visual characteristics of the one or more source images. In this regard, additionally or alternatively, the one or more source images may be encoded in a compression format that supports frequency decomposition. In this regard, additionally or alternatively, the compression format may be one of JPEG XR, JPEG2000, and AV1. In this regard, additionally or alternatively, the spatial component may be generated via at least one of manual input or a machine learning algorithm trained on labeled visual features. In this aspect, additionally or alternatively, when generating multiple fragments, multiple frequency components can be generated first, and then multiple spatial components can be generated for each of the multiple frequency components. In this aspect, additionally or alternatively, the priority order can be determined by: performing an operation on each component of the return indication, applying an application sensitivity algorithm to add noise to each component, or performing quality degradation. In this aspect, additionally or alternatively, multiple source images can be filtered to select a subset of the multiple source images for segmentation and transmission. In this aspect, additionally or alternatively, the subset of the multiple source images can be an image of a target object for analysis.

[0072] Another aspect provides a computing device, comprising: a logic subsystem including one or more processors; a memory storing instructions executable by the logic subsystem to: obtain one or more source images; segment the one or more source images to generate a plurality of fragments; determine a priority order for the plurality of fragments; and transmit the plurality of fragments to a remote computing device in the priority order, the plurality of fragments being spatial components generated by spatial decomposition of the one or more source images and / or frequency components generated by frequency decomposition of the one or more source images. In this aspect, additionally or alternatively, the frequency decomposition may be a degradation of image quality. In this aspect, additionally or alternatively, the frequency decomposition may be a decomposition of frequencies of visual characteristics of the one or more source images. In this aspect, additionally or alternatively, the one or more source images may be encoded in a compressed format that supports frequency decomposition. In this aspect, additionally or alternatively, the spatial components may be generated via at least one of manual input or a machine learning algorithm trained on labeled visual features. In this aspect, additionally or alternatively, when generating the plurality of fragments, the plurality of frequency components may be generated first, and then the plurality of spatial components may be generated for each of the plurality of frequency components. In this aspect, additionally or alternatively, the priority order can be determined by performing an operation on each component of the return indication, applying an application sensitivity algorithm to add noise to each component, or performing quality degradation. In this aspect, additionally or alternatively, the plurality of source images can be filtered to select a subset of the plurality of source images for segmentation and transmission. In this aspect, additionally or alternatively, the subset of the plurality of source images can be an image of a target object for analysis.

[0073] On the other hand, a computing device is provided, including: a logic subsystem including one or more processors; a memory storing instructions, which are executable by the logic subsystem to: obtain one or more audio data; split the one or more audio data to generate multiple fragments; determine a priority order for the multiple fragments; and transmit the multiple fragments to a remote computing device in the priority order, the multiple fragments being spatial components generated by spatial decomposition of one or more source images and / or frequency components generated by frequency decomposition of one or more source images.

[0074] It should be understood that the configuration and / or method described herein is exemplary in nature, and these specific embodiments or examples should not be considered to have limiting significance, because many variations are possible. The specific routine or method described herein can represent one or more processing strategies in any number of processing strategies. In this way, the various actions illustrated and / or described can be performed in the order illustrated and / or described, performed in other orders, performed in parallel, or omitted. Similarly, the order of the above process can be changed.

[0075] The subject matter of the present disclosure includes all novel and nonobvious combinations and subcombinations of the various processes, systems and configurations, and other features, functions, acts, and / or properties disclosed herein, as well as any and all equivalents.

Claims

1. A method for image segmentation and transmission, comprising: Obtaining one or more source images as image data at an edge device; segmenting the one or more source images to generate a plurality of segments; determining a priority order for the plurality of segments based on a measure of the effect of noise or quality degradation on objects in the one or more source images; as well as transmitting the plurality of segments to a remote computing device in the priority order, wherein The plurality of segments are generated as components, the components comprising spatial components generated by spatial decomposition of the one or more source images, and / or frequency components generated by frequency decomposition of the one or more source images; as well as The priority order is determined in the following manner: adding the noise or performing the quality degradation on the component; as well as The measure of the effect of the noise or the quality degradation on the object in the one or more source images is obtained.

2. The method according to claim 1, wherein The components include the frequency components, and The frequency decomposition is a degradation of image quality.

3. The method according to claim 1, wherein The components include the frequency components, and The frequency decomposition is a decomposition of frequencies of visual characteristics of the one or more source images.

4. The method according to claim 1, wherein The components include the frequency components, and The one or more source images are encoded in a compression format that supports the frequency decomposition.

5. The method according to claim 4, wherein The compression format is one of the following: JPEG XR, JPEG 2000, and AV1.

6. The method according to claim 1, wherein The components include the spatial components, and The spatial component is generated via at least one of: manual input or a machine learning algorithm trained on labeled visual features.

7. The method according to claim 1, wherein The components include the spatial components and the frequency components, and In generating the plurality of segments, the frequency components are first generated, and then the spatial components for each of the frequency components are generated.

8. The method according to claim 1, further comprising: An application sensitivity algorithm is applied to add the noise or perform the quality degradation to each component.

9. The method according to claim 1, wherein The plurality of source images are filtered to select a subset of the plurality of source images to segment and transmit.

10. The method according to claim 9, wherein The subset of the plurality of source images are images of a target object for analysis.

11. A computing device comprising: a logic subsystem, including one or more processors; as well as A memory storing instructions executable by the logic subsystem to: obtaining one or more source images; segmenting the one or more source images to generate a plurality of segments; determining a priority order for the plurality of segments based on a measure of the effect of noise or quality degradation on objects in the one or more source images; as well as transmitting the plurality of segments to a remote computing device in the priority order, wherein The plurality of segments are components, the components comprising spatial components generated by spatial decomposition of the one or more source images, and / or frequency components generated by frequency decomposition of the one or more source images; as well as The priority order is determined in the following manner: adding the noise or performing the quality degradation on the component; as well as The measure of the effect of the noise or the quality degradation on the object in the one or more source images is obtained.

12. The computing device of claim 11, wherein The components include the frequency components; and The frequency decomposition is a degradation of image quality.

13. The computing device of claim 11, wherein The components include the frequency components, and The frequency decomposition is a decomposition of frequencies of visual characteristics of the one or more source images.

14. The computing device of claim 11, wherein The components include the frequency components, and The one or more source images are encoded in a compression format that supports the frequency decomposition.

15. The computing device of claim 11, wherein The components include the spatial components, and The spatial component is generated via at least one of: manual input or a machine learning algorithm trained on labeled visual features.

16. The computing device of claim 11, wherein The components include the spatial components and the frequency components, and In generating the plurality of segments, the frequency components are first generated, and then the spatial components for each of the frequency components are generated.

17. The computing device of claim 11, wherein Application sensitivity algorithms are applied to add the noise or perform the quality degradation to each component.

18. The computing device of claim 11, wherein The plurality of source images are filtered to select a subset of the plurality of source images for segmentation and transmission.

19. The computing device of claim 18, wherein The subset of the plurality of source images are images of a target object for analysis.

20. A computing device comprising: a logic subsystem, including one or more processors; as well as A memory storing instructions executable by the logic subsystem to: obtaining one or more audio data; segmenting the one or more audio data to generate a plurality of segments; determining a priority order for the plurality of segments based on a measure of the effects of noise or quality degradation in the one or more audio data; as well as transmitting the plurality of segments to a remote computing device in the priority order, wherein The plurality of segments are frequency components generated by frequency components generated by frequency decomposition of the one or more audio data; as well as The priority order is determined in the following manner: adding the noise or performing the quality degradation on the frequency components; and obtaining the measure of the impact of the noise or the quality degradation in the one or more audio data.

Images