Remote partitioning under limited computing resources and rate constraints

Abstract

A vehicle communication and control system includes a first vehicle in signal communication with a remote computing system and / or a second vehicle. The first vehicle includes a sensor configured to capture an original image having a first image volume and including at least one target object. An image encoder included in the vehicle converts the original image into a masked image having a second image volume that is less than the first image volume. A segmentation unit included in the remote computing system and / or the second vehicle determines the at least one target object from the masked image, generates a masked segmentation image including a sparse segmentation of the at least one target object, and converts the sparse segmentation of the at least one target object into at least one recovered segmented object indicative of the at least one target object.

Description

TECHNICAL FIELD

[0001] The subject disclosure relates to digital image processing, and more specifically, to a vehicle image segmentation system. BACKGROUND

[0002] Modern vehicles are collecting and processing an increasing amount of data. On the other hand, vehicles have a limited view of their surroundings. Information sharing between vehicles and / or other systems, such as infrastructure or municipal traffic information servers, can enhance their understanding of the surrounding environment and improve the performance of these systems. There are two main methods to facilitate information sharing. The first method, referred to as the “direct method,” involves a vehicle / infrastructure or other information source communicating information to a relevant party, i.e., another vehicle. This method is commonly referred to as V2V or V2I. The second method, referred to as the “indirect method,” involves collecting information in a cloud or edge computing system, where the information is processed and then disseminated to vehicles of interest. This method is commonly referred to as V2N, as the information is passed from the vehicle to the network. In both cases, since the shared information is used to control and coordinate vehicle operation, the information must be processed quickly, and the processed results must be shared and exchanged with low latency.

[0003] The information exchanged using V2V, V2I, and / or V2N often includes image data. One method of processing image data for vehicle control is to implement an image segmentation system in the vehicle. Image segmentation is the process of generating a low volume segmented image from a high resolution raw image. The raw image includes various types of target objects (e.g., pedestrians, vehicles, streets, etc.) that exist in the vehicle’s surrounding environment. The resulting segmented image includes one or more segments that collectively represent the raw image, including the target objects. Thus, a vehicle’s computing system can identify boundaries from the segmented objects to determine the target objects included in the raw image data, and can make driving decisions based on the location and / or type of the identified target objects. SUMMARY

[0004] According to non-limiting embodiments, a vehicle communication and control system includes a first vehicle in signal communication with one or both of a remote computing system and a second vehicle. The first vehicle includes a sensor structured to capture an original image having a first image volume and including at least one target object. An image encoder is included in the first vehicle and structured to convert the original image into a masked image having a second image volume that is less than the first image volume. A segmentation unit is included in one or both of the remote computing system and the second vehicle. The segmentation unit is structured to determine the at least one target object from the masked image to generate a masked segmentation image including a sparse segmentation of the at least one target object and to convert the sparse segmentation of the at least one target object into at least one recovered segmented object indicative of the at least one target object.

[0005] The system further includes a feature in which the segmentation unit includes a convolutional neural network (CNN) and an image decoder. The CNN is structured to generate the sparse segmentation of the at least one target object by applying labels to pixels associated with the at least one target object and excluding pixels not associated with the at least one target object. The image decoder is structured to generate the at least one recovered segmented object based on the labels applied to the pixels.

[0006] The system further includes a feature in which the image encoder generates the masked image according to a random pixel mask and the image decoder generates the recovered segmented object according to the random pixel mask.

[0007] The system further includes a feature in which the at least one recovered segmented object is an approximation of the at least one target object included in the original image.

[0008] The system further includes a feature in which the at least one target object includes a first type of target object and a second type of target object different from the first type of target object, and in which the CNN applies a first label to pixels associated with the first type of target object and a second label to pixels associated with the second type of target object different from the first label.

[0009] The system further includes a feature in which the image decoder prioritizes decoding of pixels associated with the first label over decoding of pixels associated with a different second label.

[0010] The system further includes a feature in which the image decoder applies a matrix completion algorithm to the sparse segmentation of the at least one target object to generate the recovered segmented object.

[0011] According to another non-limiting embodiment, a remote image segmentation system includes an image encoder, a convolutional neural network (CNN), and an image decoder. The image encoder is included on a vehicle and is structured to receive an original image having a first image volume and including at least one target object, and to generate a masked image having a second image volume that is less than the first image volume. The CNN is located remotely from the image encoder. The CNN is structured to determine the at least one target object from the masked image and to generate a masked segmentation image including a sparse segmentation of the at least one target object. The image decoder is located remotely from the image encoder and is structured to convert the sparse segmentation of the at least one target object into at least one recovered segmented target object that is indicative of the at least one target object.

[0012] The remote image segmentation system further includes a feature wherein the at least one recovered segmented target object is an approximation of the at least one target object included in the original image.

[0013] The remote image segmentation system further includes a feature wherein the image encoder generates the masked image according to a random mask and the image decoder generates the at least one recovered segmented target object according to the random mask.

[0014] The remote image segmentation system further includes a feature wherein the CNN applies labels to pixels included in the masked image to produce the masked segmentation image, and wherein the image decoder generates the at least one recovered segmented target object based on the labels applied to the pixels.

[0015] The remote image segmentation system further includes a feature wherein the at least one target object includes a first type of target object and a second type of target object that is different from the first type of target object, and wherein the CNN applies a first label to pixels associated with the first type of target object and a second label to pixels associated with the second type of target object that is different from the first label.

[0016] The remote image segmentation system further includes a feature wherein the image decoder prioritizes decoding of pixels associated with the first label over decoding of pixels associated with the different second label.

[0017] The remote image segmentation system further includes a feature wherein the image decoder applies a matrix completion algorithm to the sparse segmentation of the at least one target object to generate the at least one recovered segmented target object.

[0018] According to yet another non-limiting embodiment, a method of exchanging data with a vehicle is provided. The method includes capturing, via a sensor included on the vehicle, an original image having a first image volume and including at least one target object, and generating, via an image encoder included on the vehicle, a masked image having a second image volume that is less than the first image volume. The method further includes transmitting the masked image to a convolutional neural network (CNN) located remotely from the vehicle, determining, via the CNN, the at least one target object from the masked image, and generating a masked segmentation image including a sparse segmentation of the at least one target object. The method further includes converting, via an image decoder located remotely from the vehicle, the sparse segmentation of the at least one target object into at least one recovered segmented target object that is indicative of the at least one target object.

[0019] The method further includes a feature wherein the at least one recovered segmented target object is an approximation based on a segmentation of the at least one target object of the original data.

[0020] The method further includes generating the masked image according to a random mask, and generating the at least one recovered segmented target object according to the random mask.

[0021] The method further includes applying, via the CNN, a label to pixels included in the masked image to produce a masked segmentation image, and generating, via the image decoder, the at least one recovered segmented target object based on the masked segmentation image.

[0022] The method further includes applying, via the CNN, a first label to pixels associated with a first type of target object, and a different second label to pixels associated with a second type of target object that is different than the first type of target object.

[0023] The method further includes prioritizing decoding of pixels associated with the first label over decoding of pixels associated with the different second label.

[0024] The above features and advantages of the present disclosure, and other features and advantages, will become apparent to those skilled in the art from the following detailed description, which, when taken in conjunction with the drawings, discloses various embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0025] Other features, aspects, and details of the present disclosure will become apparent from the following detailed description, which, when taken in conjunction with the drawings, discloses various embodiments.

[0026] Figure 1 is a schematic diagram of a vehicle communication and control system according to one or more embodiments;

[0027] Figure 2 depicts a remote image segmentation system according to one or more embodiments;

[0028] Figure 3a process of generating a recovered segmented image based on raw image data captured by a vehicle is depicted in accordance with one or more embodiments; and

[0029] Figure 4 is a flowchart illustrating a method of exchanging data captured by a vehicle in accordance with one or more embodiments. DETAILED DESCRIPTION

[0030] The following description is merely exemplary in nature and is not intended to limit the present disclosure, its application, or uses. It should be understood that throughout the drawings, corresponding reference numerals indicate like or corresponding parts and features.

[0031] As described herein, implementing image segmentation in a vehicle can allow for distinguishing between different types of objects (vehicles, pedestrians, etc.) in captured raw image data. However, the capture, processing, and analysis of image data representative of a vehicle’s surrounding environment (e.g., current traffic conditions) presents challenges in terms of processing and exchanging such large amounts of data, and doing so in a fast, energy-efficient, and cost-effective manner. In particular, sharing raw data captured by a vehicle’s sensors requires a high compression rate. As such, performing image encoding and decoding using a vehicle’s local computing system results in a significant loss of image quality, a decrease in image detection capabilities, and an increase in latency.

[0032] The various non-limiting embodiments described herein introduce a remote vehicle image segmentation system that requires reduced computational power to achieve reduced data exchange rates and latency metrics required to perform image segmentation on image data representative of a vehicle’s surrounding environment. The remote vehicle image segmentation system includes an image encoder, a lightweight convolutional neural network (CNN), and an image decoder. The image encoder is used locally in a vehicle computing system. However, the CNN and the image decoder are used in a remote computing system, such as a cloud computing system and / or a remote vehicle.

[0033] A vehicle can encode a raw captured high-resolution image (referred to herein as a “raw image”) that includes one or more target objects. The generated encoded image is a low-volume masked image that is then offloaded from the vehicle and transmitted to a CNN located remotely. The low-volume image described herein refers to an image that includes a subset or reduced set of pixel “groups” as compared to the total number of pixels included in the raw image. The reduced set of pixel groups is selected according to a randomized pixel mask that is programmed in the encoder and applied to the raw image to generate the masked image.

[0034] The CNN identifies one or more target objects from the masked image and generates a resulting masked image segmentation. The masked image segmentation includes one or more sparse segmentations of the target objects. The masked image segmentation is communicated to an image decoder that applies a matrix completion algorithm to the sparse segmentations to recover a segmented image representing the one or more target objects included in the original image. The low volume of the encoded masked image not only requires low processing power compared to the processing power required to process the original image, but also allows for the exchange of the masked image data with a remote computer system with low latency.

[0035] Figure 1 An operating environment including a vehicle communication and control system 10 for one or more vehicles 12a...12n (collectively motor vehicles 12a-12n) is illustratively shown. The communication and control system 10 of the vehicles 12a-12n generally includes one or more wireless carrier systems 60, satellite communication systems 68, a remote computing system 62, and / or a wireless device 57 including a display 59.

[0036] Figure 1 The vehicles 12a-12n, illustratively shown in the middle, include a propulsion system 13, which in various embodiments can include an internal combustion engine, an electric machine such as a traction motor, and / or a fuel cell propulsion system. The vehicles 12a-12n are depicted in the illustrated embodiment as passenger cars, but it will be appreciated that any other vehicle can also be used, including motorcycles, trucks, sport utility vehicles (SUVs), recreational vehicles (RVs), autonomous vehicles, watercraft, aircraft, and the like.

[0037] The vehicles 12a-12n also include a transmission 14 configured to transmit power from the propulsion system 13 to a plurality of wheels 15 according to selectable speed ratios. According to various embodiments, the transmission 14 can include a stepped-ratio automatic transmission, a continuously variable transmission, or other suitable transmission. The vehicles 12a-12n also include wheel brakes 17 configured to provide braking torque to the wheels 15. In various embodiments, the wheel brakes 17 can include friction brakes, regenerative braking systems such as electric machines, and / or other suitable braking systems.

[0038] The vehicles 12a-12n also include a steering system 16. While depicted as including a steering wheel for illustrative purposes, the steering system 16 can not include a steering wheel in some embodiments contemplated within the scope of the present disclosure.

[0039] The vehicles 12a-12n include a wireless communication system 28 configured to communicate wirelessly 58 with other vehicles (“V2V”), private infrastructure (“V2I”), or public networks (“V2N”). In an exemplary embodiment, the wireless communication system 28 is configured to communicate via a wireless local area network (WLAN) using the IEEE 802.11 standard or by using cellular data communication. Various wireless communication protocols can also be used to facilitate V2V, V2I, and / or V2N data communication, including but not limited to Dedicated Short Range Communications (DSRC), such as the Institute of Electrical and Electronics Engineers (IEEE) protocol 802.11p and / or cellular V2X (C-V2X) based on the Long Term Evolution (LTE) wireless broadband standard.

[0040] The propulsion system 13, transmission 14, steering system 16, and wheel brakes 17 are in communication with or under the control of at least one vehicle controller 22. While depicted as a single unit for purposes of illustration, the vehicle controller 22 can additionally include one or more other controllers, collectively referred to as “controllers.” The vehicle controller 22 can include a microprocessor, such as a central processing unit (CPU) or a graphics processing unit (GPU), in communication with various types of computer-readable storage devices or media. The computer-readable storage devices or media can include various types of memory, such as volatile and non-volatile memory in read-only memory (ROM), random access memory (RAM), and keep-alive memory (KAM). KAM is a persistent or non-volatile memory that can be used to store various operating variables at power down of the central processor. The computer-readable storage devices or media can be implemented using any of a number of known memory devices, such as a PROM (programmable read-only memory), EPROM (erasable PROM), EEPROM (electrically erasable PROM), flash memory, or any other electrical, magnetic, optical, or combination memory device capable of storing data, some of which represent executable instructions, used by the vehicle controller 22 in controlling the respective vehicle 12a-12n.

[0041] The vehicle controller 22 includes an autonomous driving system (ADS) 24 for automatically controlling various actuators in the vehicles 12a-12n. In an exemplary embodiment, the ADS 24 is configured to control the propulsion system 13, transmission 14, steering system 16, and wheel brakes 17 to control vehicle acceleration, steering, and braking, respectively, via a plurality of actuators 30 without human intervention in response to input from one or more sensors 26. The sensors 26 can include, but are not limited to, GPS, RADAR, LIDAR, optical cameras, thermal cameras, ultrasonic sensors, and / or suitable additional sensors.

[0042] The vehicle controller 22 also includes an image encoder 100. The image encoder 100 is structured to convert a high-resolution raw image captured by one or more sensors 26 into a masked image. When generating the masked image, the image encoder 100 takes only the values of a predefined group of pixels from the original raw image. This predefined group is determined according to a random pixel mask applied by the encoder. Pixels excluded from the predefined group are discarded, such that the resulting masked image has a substantially lower volume (i.e., substantially fewer pixels) than the original raw image.

[0043] The time-series encoding operation includes generating a time series having various data points corresponding to the masked image, thereby producing a low-volume, randomly pixelated masked image. Thus, the masked image can be exchanged via wireless communication 58 at a substantially lower bandwidth and a substantially lower latency as compared to the large bandwidth and high latency results that occur when transmitting a non-encoded image or even a fully encoded image wirelessly with very low distortion (standard compression algorithms that recover the entire original raw image).

[0044] In one or more embodiments, the masked image can be offloaded from the vehicle 12a-12n via the wireless communication system 28 and transmitted to the remote computing system 62 and / or a remotely located vehicle 12n, where it is further processed by a convolutional neural network (CNN) 102 and an image decoder 104. In one or more non-limiting embodiments, the CNN 102 and the image decoder 104 can be integrated together as a single segmentation unit 105 that is positioned remotely from the vehicle that captures the original raw image. For example, the segmentation unit 105 including the CNN 102 and the image decoder 104 can be located in the remote computing system 62 and / or a second vehicle 12n located remotely. Thus, the image encoder 100, the CNN 102, and the image decoder 104 effectively operate together as a remote image segmentation system, which will be discussed in greater detail herein.

[0045] The wireless carrier system 60 is preferably a cellular telephone system that includes one or more cellular towers 70 and any other network components necessary to connect the wireless carrier system 60 with the remote computing system 62. Each cellular tower 70 includes a transmitting-receiving antenna and a base station. The wireless carrier system 60 can implement any suitable communication technology, including for example, digital technologies such as CDMA (e.g., CDMA2000), LTE (e.g., 4G LTE or 5G LTE), GSM / GPRS, or other current or emerging wireless technologies. Other cellular tower / base station / MSC arrangements are possible and can be used with the wireless carrier system 60. For example, the base stations and cellular towers can be co-located at the same site, or they can be remote from one another, each base station can be responsible for a single cellular tower, or a single base station can serve multiple cellular towers, or multiple base stations can be coupled to a single MSC, to name a few possible arrangements.

[0046] In addition to using the wireless carrier system 60, wireless communication 58 in the form of satellite communication 68 can be used to provide one-way or two-way communication with the vehicles 12a-12n. This can be accomplished using one or more communication satellites 66 and uplink transmission stations 67. One-way communication can include, for example, satellite radio services in which program content (news, music, etc.) is received by the uplink transmission stations 67, packaged for upload, and then transmitted to the satellites 66, which broadcast the program to users. Two-way communication can include, for example, satellite telephone services that use the satellites 66 to relay telephone communications between the vehicles 12a-12n and the uplink transmission stations 67. In addition to or in place of the wireless carrier system 60, satellite telephones can be used.

[0047] The remote computing system 62 is in signal communication with the wireless carrier system 60, the satellite communication system 68, and / or the vehicles 12a-12n. In one or more non-limiting embodiments, the remote computing system 62 is implemented using an edge computing infrastructure and / or a cloud computing infrastructure, either of which is positioned remotely from the vehicles 12a-12n.

[0048] The remote computing system 62 includes one or more networked computer servers 64. The networked computer servers 64 include one or more server controllers 65. The server controllers 65 include memory and a processor configured to execute algorithms and computer readable program instructions stored in the memory.

[0049] The computer servers 64 can be accessible via a private or public network, such as the Internet, allowing them to facilitate Internet connectivity via a DNS server or other network address server that assigns IP addresses to the vehicles 12a-12n using DHCP or other suitable protocol. In an exemplary embodiment, the computer servers 64 can be configured as web servers that are accessible by the vehicles 12a-12n via data exchange between the vehicles' wireless communication systems 28 and the wireless carrier system 60.

[0050] The computer servers 64 can process a variety of data, including but not limited to diagnostic information and other vehicle data. The vehicle data can be provided by the vehicles 12a-12n via the wireless communication systems 28. The computer servers 64 can also maintain searchable databases. Thus, the computer servers 64 can operate as a database management system that allows for the input, removal, and modification of data, as well as receiving requests to locate data within the databases.

[0051] As described herein, the remote computing system 62 and / or the remote vehicle 12n can include a CNN 102 and an image decoder 104. The CNN 102 and the image decoder 104 work together with the image encoder 100 to provide a remote image segmentation system. Unlike traditional image segmentation systems, the remote image segmentation system described herein helps overcome the limited computational resources, high bandwidth costs, and low latency constraints traditionally experienced in vehicle-to-vehicle (V2V) architectures, vehicle-to-infrastructure (V2I) architectures, and / or vehicle-to-network (V2N) architectures.

[0052] More specifically, the CNN 102 is trained (e.g., offline) using a set or “pair” of pre-defined masked images and their corresponding segmented masked images (commonly referred to as “ground-truth”). In one or more non-limiting embodiments, for example, a set of 50,000 modeled masked images can be used to train the CNN 102. Once trained, the CNN 102 can receive a low-volume masked image provided by the image encoder 100, identify one or more target objects included in the low-volume masked image, and automatically construct a segmented masked image. In one or more non-limiting embodiments, the CNN 102 assigns a unique label to each pixel in the masked image associated with a target object. In this manner, individual segmented target objects can be defined based on the labels assigned to their respective pixels.

[0053] The image decoder 104 receives the masked segmented image and applies a matrix completion algorithm that reconstructs each target object to produce a recovered segmented image representing the target objects included in the original image. The recovered segmented image is an image that includes segmented target objects that are approximations of the target objects included in the original image captured by the vehicles 12a-12n. In one or more non-limiting embodiments, the decoder 104 performs a low-rank matrix operation (also referred to as a “low-rank approximation”) based on the labeled pixels of the masked segmented image to produce the recovered segmented image. For example, the decoder 104 can construct a zero-one matrix where a value of “1” is assigned to the labeled pixels and a value of “0” is assigned to all unlabeled pixels. The solution is a set of low-rank matrices equal in size to the number of labels. The decoder 104 can individually decode each low-rank matrix. The decoder 104 can then combine all the resulting completions to generate the recovered segmented image, which is an approximation of the segmentation of the original image captured by the vehicles 12a-12n.

[0054] Turning now to Figure 2According to non-limiting embodiments, a remote image segmentation system 200 is shown. As described herein, the remote image segmentation system 200 includes an image encoder 100, a CNN 102, and an image decoder 104. The image encoder 100 is located in a first vehicle 12a, while the CNN 102 and the image decoder 104 are included in a segmentation unit 105, which can be located in a remote computing system 62 and / or a second vehicle 12n. Data exchange between the image encoder 100 of the first vehicle 12a and the segmentation unit 105 is facilitated via wireless communication 58.

[0055] The image encoder 100 can be structured as an electronic hardware controller that includes a memory and a processor structured to execute encoding algorithms, image segmentation algorithms, and other computer-readable program instructions stored in the memory. Alternatively, the image encoder 100 can be implemented in the vehicle controller 22 as an integrated controller or integrated CPU.

[0056] The image encoder 100 is in signal communication with one or more vehicle sensors 26. The sensors 26 are structured to capture an initial image or series of image frames that appear in a field of view (FOV) 202 of the respective sensor. In this example, the FOV 202 of the sensor captures target objects 204a, 204b, and 204c. The target objects 204a and 204b (collectively, objects 204a-204b) are motor vehicles, while the target object 204c is a bicyclist, referred to herein as “biker” 204c. The sensors 26 output the captured initial image or image frames as a high-resolution digital image signal or “raw image” 300 (see Figure 3 ), which is received by the vehicle controller 22 and passed to the image encoder 100.

[0057] As described herein, the image encoder 100 converts the raw image 300 into a masked image 302 (see Figure 3 ) having a reduced volume and data size. In one or more embodiments, the image encoder 100 sets a code rate and applies a random masking scheme that operates according to the set code rate. The random masking scheme refers to sampling random dispersion of pixels or data points, as opposed to sampling pixels or data points that are a fixed or particular distance apart from one another. The code rate also sets the size or rate at which the raw image 300 is compressed. For example, the image encoder 100 can set a code rate of 10%, which correlates to a compression ratio of 1:10.

[0058] The random masking scheme uses the set code rate to generate a random pixel mask based on a random mask seed that is used by the image encoder 100 and the image decoder 104 to perform their respective encoding and decoding operations. For a given code rate (R), the dimensions of the randomization mask seed can be set using the following equation:

[0059] R = k / n, Equation 1

[0060] where (k) is the random portion of the plurality of (n) pixels included in the original image to be sampled or included in the resulting masked image.

[0061] Once the random pixel mask is generated, it can be programmed in the image encoder 100 and the image decoder 104 prior to implementation in the vehicles 12a-12n and the remote computing system 62. This allows the image encoder 100 and the image decoder 104 to each determine the location of the sampled pixels defined by the random pixel mask. In this manner, the image encoder 100 and the image decoder 104 can be“matched” to one another.

[0062] Once the masked image 302 is generated, it can be offloaded from the vehicle 12a and delivered (e.g., wirelessly 58) to the segmentation unit 105. The CNN 102 and the image decoder 104 included in the segmentation unit 105 can each be structured as an electronic hardware controller including a memory and a processor structured to execute image segmentation algorithms, machine learning and artificial intelligence (AI) algorithms, image decoding algorithms, and other computer readable program instructions stored in their respective memories.

[0063] As described herein, the CNN 102 is initially trained offline using a set of ground truth pairs of masked images and their corresponding segmentation masked images. These ground truth pairs are also generated according to the common random pixel mask described herein. As a result, the CNN 102 is trained to specifically learn and recognize the pixels set by the random mask and associated with different types of target objects that can be present in the original images captured by the first vehicle 12a. The target objects can include, but are not limited to, motor vehicles, motorcycles, trucks, bicyclists, pedestrians, animals, construction equipment, and the like. Moreover, the CNN 102 described herein is considered“lightweight” in that it does not need to learn, recognize, and process full resolution associated with the target objects in the original images, but rather produces a segmentation masked image having a significantly lower volume compared to the original image. As a result, the training time of the CNN 102 and the overall computational power required to process the masked image 302 is reduced.

[0064] Once trained, the CNN 102 is placed "online" and is configured to receive the masked image 302 generated by the image encoder 100. The CNN 102 is configured to generate unique labels that correspond to the pixels of the masked image that are associated with a given target object 204a, 204b, 204c, while excluding pixels that are not associated with the target object 204a, 204b, 204c. In one or more non-limiting embodiments, each unique label includes a unique color. For example, pixels associated with motor vehicles can be converted to red pixels, pixels associated with bicyclists can be converted to white pixels, pixels associated with pedestrians can be converted to green pixels, and so on. Thus, the CNN 102 analyzes the masked image 302, identifies pixels associated with the target objects 204a, 204b, and 204c based on its training, and applies a given label to each pixel associated with a target object included in the masked image 302. For example, with reference to Figure 2 and 3 , a first label (e.g., red pixels) can be applied to each pixel associated with the captured vehicles 204a and 204b, while a second label (e.g., white pixels) can be applied to each pixel associated with the captured bicyclist 204c.

[0065] The CNN 102 generates a masked segmented image 304 (see Figure 3 ) that includes only a portion or subset of the pixels compared to the masked image 302. Thus, the masked segmented image 304 provides a sparse segmentation of the target objects 204a, 204b, and 204c included in the original image 300. As described herein, the CNN 102 can apply unique labels to pixels associated with different types of target objects 204a, 204b, and 204c. In this manner, the image decoder 104 can preferentially decode the sparsely segmented target objects 204a, 204b, and 204c, as discussed in greater detail herein.

[0066] The image decoder 104 receives the masked segmented image 304 and applies a matrix completion algorithm to the masked segmented image 304 to generate a restored segmented image 306 (see Figure 3 ). The matrix completion algorithm operates on each label separately and essentially adds additional pixels or data points beyond those included in the segmented masked image 304 that are associated with the respective label. Thus, the restored segmented image 306 provides an approximation of the segmentation performed directly on the original image data and includes segmented target objects 204a, 204b, and 204c that are approximations of the target objects 204a, 204b, and 204c included in the original image 300 originally captured by the vehicle 12a.

[0067] In one or more non-limiting embodiments, the image decoder 104 can be programmed to prioritize decoding the target objects 204a, 204b, and 204c based on the labels (e.g., colors) of the pixels. For example, pixels that have applied thereto a label (e.g., white pixels) corresponding to the cyclist target object 204c can be determined to have a higher priority than labels (e.g., green pixels) corresponding to the vehicle target objects 204a and 204b. Based on the priority, the image decoder 104 can select to decode target object pixels having a higher priority than target object pixels having a lower priority.

[0068] Although the restored segmented image 306 is an approximation of the segmentation as compared to performing the segmentation directly on the original image 300, it can still be processed and analyzed by the server controller 65 and / or vehicle controller 22 of the second vehicle 12n (see FIG. 1) to determine the location of objects 204a, 204b, and 204c relative to the second vehicle 12n. Figure 1 ) The processed data can provide a wide variety of information, including but not limited to current traffic conditions, the location of objects 204a, 204b, and 204c not only relative to the vehicle 12a that captured the original image 300, but also relative to other vehicles 12n in the vicinity of the image capturing vehicle 12a, and the location of construction zones.

[0069] The processed data can also facilitate a“see-through” feature that allows a following vehicle to“see-through” to a leading vehicle that has already captured an original image. For example, the image segmentation of an original image captured by a leading vehicle 12a can be decoded in a remote computing system (e.g., cloud computing system 62) to determine one or more objects 204a, 204b, and 204c located ahead of the leading vehicle 12a. The server controller 65 can then determine the distance between the objects 204a, 204b, and 204c and the leading vehicle 12a and warn a following vehicle 12n of a possible stopping event in the absence of the following vehicle 12n visually being aware of the objects 204a, 204b, and 204c.

[0070] Reference is now made to Figure 4The flowchart illustrates a method of exchanging data captured by a vehicle, in accordance with one or more non-limiting embodiments. The method begins at operation 400, and at operation 402, a first vehicle 12a captures an original image 300 including one or more target objects 204a, 204b, and 204c. The original image 300 can be captured using one or more sensors 26 (e.g., cameras) installed on the first vehicle 12a. At operation 404, an image encoder 100 installed on the first vehicle 12a converts the original image 300 into a masked image 302. At operation 406, the masked image 302 is offloaded from the first vehicle 12a to a CNN 102. The CNN 102 can be located in a remote computing system 62 (e.g., a cloud computing network) and / or a second vehicle 12n located remotely from the first vehicle 12a. At operation 408, the CNN 102 generates a masked segmented image 304 based on the masked image 302. At operation 410, an image decoder 104 generates a recovered segmented image 306 based on the masked segmented image 304. The recovered segmented image 306 includes one or more segmented target objects 204a, 204b, 204c, which are approximations of the target objects 204a, 204b, 204c included in the original image 300. Accordingly, the method ends at operation 412.

[0071] Various embodiments are described herein with reference to the related drawings. Alternative embodiments can be devised without departing from the scope of the present teachings. In the following description, various connections and positional relationships are set forth in the description and / or shown in the drawings. Unless otherwise indicated, the connections and / or positional relationships can be direct or indirect, and the present description is not intended to exclude any equivalent alternative(s) having differently configured or arranged elements. Thus, the term "connected" can mean directly or indirectly connected, and the term "positioned on" can mean directly positioned on or indirectly positioned on, unless otherwise indicated. In addition, various tasks and process steps described herein can be combined, rearranged, or otherwise re-sequenced without departing from the teachings of the present description.

[0072] The following definitions and abbreviations are used in the present application. As used herein, the terms "comprises," "comprising," "includes," "including," "has," "having," "contains," "containing," or any other variation thereof, are intended to cover a non-exclusive inclusion. For example, a composition, a mixture, a process, a method, an article, or an apparatus that comprises a list of elements is not necessarily limited to only those elements but can include other elements not expressly listed or inherent to such composition, mixture, process, method, article, or apparatus.

[0073] Furthermore, the term“exemplary” is used herein to mean“serving as an example, instance, or illustration.” Any embodiment or design described herein as“exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments or designs. The terms“at least one” and“one or more” can convey the inclusion of any integer number of items, i.e., from one to an unlimited number of items, including, for example, one, two, three, four, etc. In addition, the terms“multiple,”“plurality,” or the like can convey the inclusion of any integer number of items, i.e., two, three, four, five, etc. The term“coupled” can include an indirect“coupling” and a direct“coupling”

[0074] The terms“approximately,”“substantially,”“about,” and variations thereof, are intended to encompass amounts that would be considered, by one of ordinary skill in the art, as being equivalent in context to the stated circumstance. For example,“about” can include a range of ± 8% or 5%, or 2% of a given value.

[0075] In the interest of clarity, not all of the traditional technology related to the fabrication and use of embodiments can be described in detail herein. In particular, various aspects of the computing systems, algorithms, and specific computer programs implementing the various techniques described herein are well known. Accordingly, in the interest of brevity, a number of traditional implementation details will not be described here in detail or will be omitted altogether where it is believed that such details will be of little or no interest to someone skilled in the art. Thus, the skilled artisan will understand that the technology described herein can be employed in a variety of software and hardware implementations (including adaptations, perfections, enhancements, etc.) in many potentially different contexts. All such implementations are considered to be within the scope of the present technology.

[0076] The present embodiments include systems, methods, and / or computer program products at all possible levels of integration. The computer program product can include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present technology.

[0077] The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, and any suitable combination of the foregoing. Computer readable storage media, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.

[0078] Computer readable program instructions for carrying out operations of an embodiment can be assembly-language instructions, instructions-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, configuration data for integrated circuitry, or source code and object code written in any combination of one or more programming languages, including an object-oriented programming language such as Smalltalk, C++, or the like, and procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present teachings.

[0079] Aspects of the present teachings are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to various embodiments. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer readable program instructions.

[0080] These computer readable program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions can also be stored in a computer readable storage medium that can direct a computer, a programmable data processing apparatus, and / or other devices to function in a particular manner, such that the computer readable storage medium having instructions stored therein comprises an article of manufacture including

[0081] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer, other programmable apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0082] The flow and block diagrams in the drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments. In this regard, each block in the flow and block diagrams can represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical functions. In some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks can sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block and / or flow diagrams and combinations thereof can be implemented by a dedicated hardware-based system that performs the specified functions or actions, or combinations of dedicated hardware and computer instructions.

[0083] The description of the various embodiments is intended for illustration, and is not intended to limit or exclude any embodiments, whether or not a modification is specifically recited in this description. Many modifications and variations are possible that fall within the scope of the above-described embodiments. The terms used herein are to be given their ordinary and customary meaning to a person of ordinary skill in the art and are not intended to be used to unduly restrict the scope of the described embodiments.

Claims

1. A vehicle communication and control system, comprising: a first vehicle in signal communication with one or both of a remote computing system and a second vehicle, the first vehicle including a sensor configured to capture an original image having a first image volume and including at least one target object; an image encoder included in the first vehicle and configured to convert the original image into a masked image having a second image volume that is less than the first image volume; a segmentation unit included in one or both of the remote computing system and the second vehicle, the segmentation unit configured to determine the at least one target object from the masked image, generate a masked segmentation image including a sparse segmentation of the at least one target object, and convert the sparse segmentation of the at least one target object into at least one recovered segmented target object indicative of the at least one target object; a convolutional neural network configured to generate the sparse segmentation of the at least one target object by applying a label to pixels associated with the at least one target object and excluding pixels not associated with the at least one target object; and an image decoder configured to generate the at least one recovered segmented target object based on the label applied to the pixels, wherein the image encoder generates the masked image from a random pixel mask and the image decoder generates the recovered segmented target object from the random pixel mask. The at least one recovered segmented target object is an approximation of the at least one target object included in the original image.

2. The system of claim 1, wherein, The at least one target object includes a first type of target object and a second type of target object different from the first type of target object, wherein the convolutional neural network applies a first label to pixels associated with the first type of target object and a second label to pixels associated with the second type of target object different from the first label.

3. The system of claim 2, wherein, The image decoder prioritizes decoding of pixels associated with the first label over decoding of pixels associated with the second, different label.

4. The system of claim 3, wherein, 5. A remote image segmentation system, comprising: an image encoder included on a vehicle and configured to receive an original image having a first image volume and including at least one target object, and generate a masked image having a second image volume that is less than the first image volume; a convolutional neural network located remotely from the image encoder and configured to determine the at least one target object from the masked image and generate a masked segmentation image including a sparse segmentation of the at least one target object; and an image decoder located remotely from the image encoder and configured to convert the sparse segmentation of the at least one target object into at least one recovered segmented target object indicative of the at least one target object, wherein the at least one recovered segmented target object is an approximation of the at least one target object included in the original image, and the at least one target object includes a first type of target object and a second type of target object different from the first type of target object, wherein the convolutional neural network applies a first label to pixels associated with the first type of target object and a second label to pixels associated with the second type of target object different from the first label. wherein the image encoder generates a masked image from the random mask and the image decoder generates at least one recovered segmented target object from the random mask, wherein the convolutional neural network applies labels to pixels included in the masked image to produce the masked segmentation image, and wherein the image decoder generates the at least one recovered segmented target object based on the labels applied to the pixels.

6. The remote image segmentation system of claim 5, wherein, The at least one target object includes a first type of target object and a second type of target object different from the first type of target object, wherein the convolutional neural network applies a first label to pixels associated with the first type of target object and a second label to pixels associated with the second type of target object different from the first label, and wherein the image decoder prioritizes decoding of the pixels associated with the first label over decoding of the pixels associated with the second label different.

7. The remote image segmentation system of claim 5, wherein, The image decoder applies a matrix completion algorithm to the sparse segmentation of the at least one target object to generate the at least one recovered segmented target object.

8. A method of exchanging data with a vehicle, the method comprising: capturing an original image via a sensor included on the vehicle, the original image having a first image volume and including at least one target object; generating, via an image encoder included on the vehicle, a masked image having a second image volume smaller than the first image volume; transmitting the masked image to a convolutional neural network located remotely from the vehicle; determining, via the convolutional neural network, the at least one target object from the masked image, generating a masked segmentation image including a sparse segmentation of the at least one target object; and converting, via an image decoder located remotely from the vehicle, the sparse segmentation of the at least one target object into at least one recovered segmented target object indicative of the at least one target object, wherein the convolutional neural network applies labels to pixels included in the masked image to produce the masked segmentation image, and wherein the image decoder generates the at least one recovered segmented target object based on the labels applied to the pixels.

Images