Data compression system and method of use

Abstract

The system comprises a non-transitory computer-readable medium configured to store instructions and a processor coupled to the non-transitory computer-readable medium, the processor configured to execute instructions for generating a mask based on received data from a sensor, the mask including a plurality of importance values, each region of the received data being assigned to a corresponding importance value of the plurality of importance values, encoding the received data based on the mask, transmitting the encoded data to a decoder to define reconstructed data, calculating a loss based on the reconstructed data, the received data, and the mask, and providing training to an encoder for encoding the received data based on the calculated loss.

Description

[Technical field]

[0001] FIELD OF THE DISCLOSURE The present disclosure relates to a data compression system and a method for using the data compression system. [Background technology]

[0002] Autonomous driving involves collecting data from sensors, processing the data, and transmitting the data from any location to another. The amount of data collected by the sensors is quite large. As a result, tree-based representations are used in some systems to help reduce the amount of data that is transmitted and processed. Data compression techniques in some systems compress all of the data received from the sensors.

[0003] An example of a type of sensor used to collect data includes a light detection and ranging (LiDAR) sensor. A LiDAR sensor emits a beam of light at a predefined interval. The emitted beam contacts an object and is reflected back to the sensor. Based on the time between emission and detection, the LiDAR sensor can determine the distance between the object and the sensor. Summary of the Invention

[0004] According to a first exemplary aspect of the present disclosure, a system is provided that includes a non-transitory computer-readable medium configured to store instructions and a processor coupled to the non-transitory computer-readable medium, the processor configured to execute instructions for generating a mask based on received data from a sensor, the mask including a plurality of importance values, each region of the received data being assigned to a corresponding importance value of the plurality of importance values, encoding the received data based on the mask, transmitting the encoded data to a decoder to define reconstructed data, calculating a loss based on the reconstructed data, the received data, and the mask, and providing training to an encoder for encoding the received data based on the calculated loss.

[0005] According to a second exemplary aspect of the present disclosure, a method is provided, the method including: generating a mask using a processor based on received data from a sensor, the mask including a plurality of importance values, each region of the received data being assigned to a corresponding importance value of the plurality of importance values; encoding the received data based on the mask; transmitting the encoded data to a decoder to define reconstructed data; calculating a loss based on the reconstructed data, the received data, and the mask; and providing training to an encoder for encoding the received data based on the calculated loss.

[0006] According to a third exemplary aspect of the present disclosure, there is provided a system, the system comprising: a sensor configured to collect data; a non-transitory computer-readable medium configured to store instructions; and a processor coupled to the non-transitory computer-readable medium. The processor is configured to execute instructions for generating a mask based on the collected data, the mask including a plurality of importance values, each region of the collected data being assigned a corresponding importance value of the plurality of importance values. The system further includes an encoder configured to receive the collected data and the generated mask. The encoder is configured to compress the collected data based on the mask and transmit the compressed data to a decoder for defining reconstruction data.

[0007] According to a fourth exemplary aspect of the present disclosure, a non-transitory computer-readable medium configured to store instructions is provided to cause a processor to generate a mask based on collected data from a sensor, the mask including a plurality of importance values, each region of the collected data being assigned to a corresponding importance value of the plurality of importance values, instruct an encoder to compress the collected data based on the generated mask, and instruct the encoder to transmit the compressed data to a decoder for defining reconstruction data. [Brief description of the drawings]

[0008] Aspects of the present disclosure can be best understood from the following detailed description when read in conjunction with the accompanying drawings, in which: It should be noted that, in accordance with standard industry practice, various features have not been drawn to scale, and in fact the dimensions of various features may be arbitrarily increased or decreased for clarity of illustration. [Figure 1] 1 is a schematic diagram of a data compression system according to some embodiments. [Diagram 2] 4 is a flow diagram of a method for compressing data according to some embodiments. [Figure 3A] 1 is a flow diagram of a method for generating a mask, according to some embodiments. [Figure 3B] 1 is a diagram of an image received by a sensor according to some embodiments. [Figure 3C] 1 is a diagram of a point cloud received by a sensor according to some embodiments. [Figure 3D] FIG. 1 is a diagram of a mask that can be applied to sensor data, according to some embodiments. [Figure 4] 1 is a flow diagram of a method for applying rules to received data according to some embodiments. [Diagram 5] 1 is a flow diagram of a method for generating a ground truth mask according to some embodiments. [Figure 6] 4 is a flow diagram of a method of using a data compression system according to some embodiments. [Figure 7] 1 is a block diagram of a system for implementing a data compression method according to some embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] The following disclosure provides many different embodiments or examples for implementing various features of the provided subject matter. Below, specific examples of components, values, operations, materials, arrangements, etc. are described to simplify the disclosure. Of course, these are merely examples and are not intended to be limiting. Other components, values, operations, materials, arrangements, etc. are contemplated. For example, the formation of a first feature above or on top of a second feature in the following description may include embodiments in which the first feature and the second feature are formed in direct contact, and may also include embodiments in which an additional feature may be formed between the first feature and the second feature such that the first feature and the second feature are not in direct contact. Furthermore, the present disclosure may repeat reference numbers and / or letters in various examples. This repetition is for simplicity and clarity and does not, in itself, dictate a relationship between the various embodiments and / or configurations described.

[0010] Additionally, spatially relative terms such as "beneath," "below," "lower," "above," "upper," and the like may be used herein for ease of description to describe the relationship of one element or feature to another element or feature as shown in the figures. The spatially relative terms are intended to encompass various orientations of the device in use or operation in addition to the orientation shown in the figures. The device may be oriented in other directions (rotated 90 degrees or in other directions) and the spatially relative descriptors used herein may be similarly interpreted accordingly. The use of the term "and / or" means that each option can be used individually or in combination with any or all of the other options.

[0011] Autonomous driving or remote monitoring of driving conditions or equipment involves receiving data from sensors and then processing and transmitting the data in a timely manner to identify and describe continuously changing conditions around the vehicle. Compressing the data received from the sensors helps reduce the transmission time of the information. Including and transmitting a large amount of data still introduces some delay time and increases the risk of erroneous data when the compressed data is decoded. Further reduction in processing and transmission time by generating a mask based on the data received by the sensors and applying the mask when transmitting the data, i.e., by an encoder, helps to increase the speed and accuracy of the data utilized for autonomous driving and remote monitoring.

[0012] Applying a mask to the transmitted data helps to remove parts of the received data or reduce the importance of parts of the received data. By targeting parts of the received data that are most likely to have the greatest impact on the vehicle, the amount of data to be processed and transmitted is reduced. As a result, the processing load, transmission load and latency are reduced. Thus, autonomous driving or remote monitoring can better respond to detected objects that are most relevant and of the highest importance. Reducing the amount of transmitted data also reduces the risk of errors in decoding the transmitted data.

[0013] In some embodiments, the mask is developed and updated using artificial intelligence (AI). Additionally, user input information may further clarify and refine the mask. In some embodiments, the mask is developed based on a combination of rules for determining which portions of the received data have a high degree of importance. By determining the variation between the data originally received and the reconstructed data after decoding of the compressed data, the machine learning generated mask and the encoding process may be updated to continually improve the performance of the data compression system.

[0014] The following focuses on light detection and ranging (LiDAR) sensors, however, those skilled in the art will appreciate that the data compression system can be used with other types of sensors, such as visible light sensors, infrared light sensors, radio detection and ranging (RADAR), sound navigation and ranging (SONAR), or other suitable sensor technologies.

[0015] FIG. 1 is a schematic diagram of a data compression system 100 according to some embodiments. The data compression system 100 is implemented using one or more processors, such as processor 702 (FIG. 7) described below. The data compression system 100 is configured to receive data 105 from at least one sensor. The data compression system 100 is further configured to receive additional information 110 from a user using, for example, an input / output (I / O) interface, such as a graphical user interface (GUI). The additional information 110 and the data 105 are used to generate a mask 120, for example, using a processor coupled to an encoder 130. The encoder 130 receives both the mask 120 and the data 105. The mask 120 is applied to the data 105 at the encoder to determine which information to transmit and in what order. The encoder 130 is configured to apply the mask to the data 105 to generate compressed data 137. The compressed data 137 is then transmitted from the encoder 130 to a decoder 140 using a network 135, for example by a transmitter. The decoder 140 reconstructs the compressed data 137, for example using a processor coupled to the decoder 140, to generate reconstructed data 150. The reconstructed data 150 can be used by an autonomous vehicle or a remote monitoring device to determine whether to adjust control of the vehicle or another remotely monitored object. Additionally, the reconstructed data 150, which combines the data 105 and the mask 120, is received by a loss calculator 160, which is used to determine a loss 165, for example using a processor coupled to the encoder 130 or the decoder 140, between the original received data 105 and the reconstructed data 150. By determining the loss 165, the mask 120, as well as the encoder 130 and the decoder 140, can be trained to improve the performance of the data compression system 100.

[0016] Further, the data compression system 100 may generate a GT mask 170 based on received ground truth (GT) segments 175, e.g., from a user, and additional information 177, e.g., using a processor coupled to the encoder 130 or the decoder 140. The GT mask 170 can be used to train the mask 120. Since the GT mask 170 is generated using known data, the GT mask 170 is error-free in the object detection results. The GT segments 175 are known object segments in the data used to develop the GT mask 170. The additional information 177 is rules applied to the GT segments 175, e.g., using a processor coupled to the encoder 130 or the decoder 140, to generate the GT mask 170. In some embodiments, the additional information 177 is the same as the additional information 110. In some embodiments, the additional information 177 is different from the additional information 110. The result of the GT mask 170 can be used by the loss calculator 160, implemented using, for example, a processor coupled to the encoder 130 or the decoder 140, to more accurately determine the loss 165. In some embodiments, the GT mask 170 is omitted. By omitting the GT mask 170, the processing load of the data compression system 100 is reduced.

[0017] Additionally, the object detector 180 of the data compression system 100 may use the reconstruction data 150 to detect objects in the reconstruction data 150. The output of the object detector 180 may be used by the loss calculator 160 to calculate additional object detection losses to further improve the accuracy of the data compression system 100. In some embodiments, the object detector 180 is omitted. Omitting the object detector 180 reduces the processing load of the data compression system 100. However, omitting the object detector 180 increases the risk that the accuracy of the training of the mask 120 will be reduced.

[0018] The data 105 includes either a three-dimensional (3D) or two-dimensional (2D) representation of an area detectable by at least one sensor. In some embodiments, the at least one sensor is mounted on a vehicle. In some embodiments, the at least one sensor is mounted on various vehicles. In some embodiments, the at least one sensor is mounted on a fixed object, such as a traffic light or road sign. In some embodiments, the data 105 is received from multiple sensors. In some embodiments, all of the sensors include the same type of sensor, e.g., LiDAR. In some embodiments, at least one of the sensors is a different type of sensor than at least one other sensor, e.g., LiDAR and a visible light sensor. In some embodiments, the data 105 includes a point cloud.

[0019] The additional information 110 includes rules for determining how the mask 120 is generated. In some embodiments, the additional information 110 is received from a user. In some embodiments, the data compression system 100 is configured to alert the user in response to the loss 165 being higher than a predetermined threshold. The loss is calculated during training of the data compression system 100. Once the data compression system is fully trained, the loss calculation is omitted. In some embodiments, the alert to the user includes an audio alert, a text alert, or another suitable alert. In some embodiments, the alert is sent to a mobile device accessible to the user. In some embodiments, the user may request an update to the training of the data compression system 100.

[0020] In some embodiments, the additional information 110 includes rules based on the distance of the object from the sensor. For example, applying one or more thresholds to the LiDAR data may aid in determining the importance level of different pixels in the received data without first performing object detection. The following description is provided as an example. Those skilled in the art will understand that additional implementations and rules are within the scope of this disclosure. A first portion of the data 105 including a first object detected by the sensor within a first threshold distance from the sensor is assigned a first importance value, e.g., a 1.0 value. A second portion of the data 105 including a second object detected by the sensor from the first threshold distance to a second threshold distance from the sensor is assigned a second importance value, e.g., a 0.5 value. A third portion of the data 105 including a third object detected by the sensor further than the second threshold distance from the sensor is assigned a third importance value, e.g., a 0 value. Each portion of the data 105 is analyzed to determine the proximity between the sensor and any detected objects within the corresponding region. Detection of an object by a sensor does not necessarily include classification of the object or a determination of the type of object, but only a determination of the presence of an object within the detected proximity of the sensor. For example, in some embodiments, the LiDAR data provides a pixel-by-pixel distance of the data from the sensor. One or more thresholds may be applied to the LiDAR data to determine the importance level of various pixels in the received data.

[0021] In some embodiments, the first distance threshold and the second distance threshold are determined by a user. In some embodiments, the first distance threshold and the second distance threshold are determined or recommended by the data compression system 100. In some embodiments, the first distance threshold and / or the second distance threshold are fixed values. In some embodiments, the first distance threshold and / or the second distance threshold vary based on various parameters known to the data compression system 100. For example, in some embodiments, the first distance threshold and / or the second distance threshold increase as the speed of the vehicle increases. In some embodiments, the first distance threshold and / or the second distance threshold decrease as the detection range of the sensor decreases. For example, during bad weather or when the sensor becomes dirty, the first distance threshold and / or the second distance threshold are increased to avoid further loss from data having a noise component that has a large influence in the data. In some embodiments, the first distance threshold and / or the second distance threshold are adjusted based on the detected path of the road. For example, the first distance threshold and / or the second distance threshold are smaller for roads with curves than for straight roads. In some embodiments, the road route is determined based on maps and data from a Global Positioning System (GPS) stored in the data compression system 100. In some embodiments, three or more distance thresholds are used. As the number of distance thresholds increases, the accuracy and precision of the mask 120 increases, however, the processing load of the data compression system increases. In some embodiments, the first distance threshold is approximately equal to 200 meters (m). In some embodiments, the first distance threshold is in the range of about 100 m to about 500 m. If the first distance threshold is too large, in some cases, the risk of the reconstruction data 150 being inaccurate decreases. If the first distance threshold is too small, in some cases, the processing load of the data compression system 100 becomes excessive. In some embodiments, the second distance threshold is approximately equal to 500 m. In some embodiments, the second distance threshold is in the range of about 300 m to about 750 m. If the second distance threshold is too large, in some cases, the processing load of the data compression system 100 becomes excessive.If the second distance threshold is too small, then there is an increased risk that the reconstruction data 150 may be inaccurate.

[0022] In some embodiments, the additional information 110 includes rules based on classification of objects detected by the sensor. The following description is provided as an example. Those skilled in the art will understand that additional implementations and rules are within the scope of the present disclosure. The classification of the object determines what type of object is detected. For example, automobile, bicycle, pedestrian, building, etc. Based on the determined type of object, an importance value is assigned to the detected object. A first object category determined based on the data 105 is assigned a first importance value, e.g., a 1.0 value. A second object classification determined based on the data 105 is assigned a second importance value, e.g., a 0.6 value. A third object classification determined based on the data 105 is assigned a third importance value, e.g., a 0.4 value. A fourth object classification determined based on the data 105 is assigned a fourth importance value, e.g., a 0 value. Each portion of the data 105 is analyzed to determine whether any object of the classification is within the corresponding region. In some embodiments, the type of object in each of the object classifications is determined by a user. In some embodiments, the type of object in each of the object classes is automatically determined based on the object's position relative to the road. For example, GPS information can be used to determine the relative position between the detected object and the road. In some embodiments, the type of object in each of the object classes is automatically determined based on the detected size of the object. In some embodiments, the type of object in each of the object classes is automatically determined based on the predicted moving speed of the detected object. For example, a detected vehicle is predicted to have a faster moving speed than a person or a bicycle. In some embodiments, various combinations of parameters are used to automatically classify the detected objects.

[0023] In some embodiments, the first object category includes vehicles, pedestrians, and / or bicycles. In some embodiments, the second object classification includes roads, sidewalks, and / or road signs. In some embodiments, the third object classification includes buildings, trees, fences, utility poles, and / or utility poles. In some embodiments, the fourth object classification includes sky and / or flying objects.

[0024] In some embodiments, the additional information 110 includes rules based on the detected speed of the detected object. The following description is provided as an example. Those skilled in the art will understand that additional implementations and rules are within the scope of the present disclosure. Using the data 105 collected at various times, the data compression system 100 may determine a change in the position of the detected object relative to the sensor. Based on the determined change in position, the data compression system 100 may determine a velocity of the detected object relative to the sensor. The velocity includes both the speed and the direction of movement of the detected object. A first object determined to be moving toward the sensor at a high speed based on the data 105 is assigned a first importance value, e.g., a 1.0 value. A second object determined to be moving toward the sensor at a low speed based on the data 105 is assigned a second importance value, e.g., a 0.6 value. A third object determined to be moving away from the sensor at a low speed based on the data 105 is assigned a third importance value, e.g., a 0.4 value. A fourth object determined to be moving away from the sensor at a high speed based on the data 105 is assigned a fourth importance value, e.g., a value of 0. Each portion of the data 105 is analyzed to determine whether any objects of the classification are within the corresponding region. A speed threshold can be used to determine whether a detected object is moving at a high speed or a low speed. A detected object determined to be moving at a speed equal to or greater than the speed threshold is determined to be moving at a high speed, and a detected object determined to be moving at a speed less than the speed threshold is determined to be moving at a low speed. In some embodiments, the speed threshold is set by a user. In some embodiments, the speed threshold is set or recommended by the data compression system 100. In some embodiments, the speed threshold is a fixed value. In some embodiments, the speed threshold is adjustable. In some embodiments, the speed threshold is adjusted based on the speed of the vehicle, including the system, detected road conditions, detected visibility conditions, or other suitable parameters. In some embodiments, the speed threshold is approximately equal to 60 kilometers per hour (km / h). In some embodiments, the speed threshold is in the range of about 20 km / h to about 100 km / h. If the velocity threshold is too large, then in some cases the risk of inaccuracies in the mask 120 increases.If the rate threshold is too small, then in some cases the processing load on the data compression system 100 may increase without any noticeable improvement in performance.

[0025] Those skilled in the art will recognize that in some embodiments, the combinations of the specific examples of rules described above can be combined in various combinations. For example, in some embodiments, a combination of object distance and classification is used to specify an importance value for a detected object. Those skilled in the art will also recognize that in some embodiments, additional or several levels of importance values ​​can be used. As the number of importance value levels increases, in some cases, the processing load of the data compression system 100 increases. As the number of importance value levels decreases, in some cases, the risk of inaccuracies in the reconstruction data 150 increases.

[0026] The mask 120 is generated based on the data 105 and the additional information 110. Generating the mask 120 includes applying the rules of the additional information 110 to the data 105 from the sensor. The data 105 is separated into regions containing one or more pixels. Each of the regions of the data 105 is assigned an importance value based on the additional information 110. As the number of regions of the data 105 increases, the processing load of the data compression system 100 increases. However, as the number of regions of the data 105 increases, the accuracy of the reconstruction data 150 improves. In some embodiments, the number of regions of the data 105 is predetermined. In some embodiments, the number of regions of the data 105 varies based on different parameters. For example, in some embodiments, the number of regions 105 decreases as the speed of the vehicle on which the sensor is mounted increases. This reduces the processing load of the data compression system 100 and reduces the delay time between the collection of the data 105 and the output of the reconstruction data 150.

[0027] The encoder 130 extracts features from the data 105 based on the mask 120. In some embodiments, portions of the data 105 corresponding to high importance values ​​in the mask 120 are extracted before portions of the data 105 having low importance values. In some embodiments, portions of the data 105 having high importance values ​​are extracted with a higher resolution than portions of the data 105 having low importance values. In some embodiments, a combination of priority extraction and high resolution is utilized by the encoder 130. In some embodiments, the encoder 130 includes a residual network, such as a ResNet-50 or a fully convolutional residual network. In some embodiments, the encoder 130 includes a dilated convolutional layer or a strided convolutional layer. The encoder 130 generates compressed data 137 that is transmitted by the network 135. In some embodiments, portions of the compressed data 137 corresponding to the data 105 having high importance values ​​are transmitted before portions of the compressed data 137 corresponding to the data 105 having low importance values.

[0028] In some embodiments, a transmitter or transceiver is utilized by the data compression system 100 to transmit the compressed data 137 over a network 135. In some embodiments, the network 135 is a wireless network. In some embodiments, the network 135 is a wired network. In some embodiments, the compressed data 137 is transmitted to multiple locations, for example, to a decoder 140 in a vehicle as well as to a remote monitoring device.

[0029] The decoder 140 reconstructs the compressed data 137 received from the network 135 to form reconstructed data 150. In some embodiments, a receiver or transceiver is utilized by the data compression system 100 to receive the compressed data 137 from the network 135. In some embodiments, the decoder 140 includes a residual network, such as a ResNet-50 or a fully convolutional residual network. In some embodiments, the decoder 140 includes a dilated convolutional layer or a strided convolutional layer.

[0030] The reconstruction data 150 is based on the data 105 and can be used by an autonomous driving system (not shown) or a remote monitoring device (not shown). In some embodiments, portions of the data 105, e.g., portions with low importance values, are missing from the reconstruction data 150. In some embodiments, portions of the reconstruction data 150, e.g., portions with low importance values, are at a lower resolution than other portions of the reconstruction data 150, e.g., portions with high importance values. In some embodiments, the reconstruction data 150 is provided as an image that can be viewed by a remote monitoring device. In some embodiments, the reconstruction data 150 includes information readable by a processor, e.g., for autonomous driving or remote monitoring.

[0031] The data compression system 100 reduces the processing load for generating the reconstruction data 150 compared to other approaches. By reducing the resolution of the data 105 or excluding portions of the data 105 in generating the reconstruction data 150, the processing load on the receiver is reduced without significantly losing relevant information in the reconstruction data 150. Reducing the processing load while preserving relevant information means that an autonomous driving system can identify hazards more quickly and accurately.

[0032] Loss calculator 160 compares reconstructed data 150 to the detected data, also using mask 120, to determine whether data 105 was correctly encoded, transmitted, and decoded. In some embodiments, loss calculator 160 determines loss 165 based on a importance-based mean squared error loss algorithm. For example, in some embodiments, loss 165 is determined using the following equation (1):

number

[0033] The loss 165 is fed back to the data compression system 100 to train the encoder 130 and the decoder 140 to improve the accuracy of the reconstructed data 150. In some embodiments, the loss 165 is used to improve the generation of the mask 120, as described below.

[0034] The GT mask 170 is generated based on known information provided by the GT segments 175 and the additional information 177. The GT mask 170 is generated similarly to the generation of the mask 120. In some embodiments, the additional information 177 is identical to the additional information 110. In some embodiments, the number of regions of the data 105 used to generate the mask 120 is equal to the number of regions applied to the GT segments 175 to generate the GT mask 170. In some embodiments, the data 105 is used to generate the GT mask 170 in order to apply known information, such as object classification, to the data 105 during the generation of the GT mask 170. The GT mask 170 is provided to the loss calculator 160 to determine the loss 165.

[0035] In some embodiments, where loss calculator 160 receives GT mask 170, loss calculator 160 determines loss 165 based on a combination of the mask loss and a importance-based mean squared error loss algorithm. For example, in some embodiments, loss 165 is determined using the following equation (2):

number

[0036] Object detector 180 is configured to detect objects based on reconstruction data 150. In some embodiments, object detector 180 classifies detected objects from reconstruction data 150. In some embodiments, object detector 180 modifies reconstruction data 150 viewable by a remote monitoring device to highlight detected objects. For example, in some embodiments, a box or other identification system is placed around or adjacent to the detected object to attract the attention of the remote monitoring device. In some embodiments, the highlighting is performed based on a classification of the detected object. For example, in some embodiments, a vehicle is highlighted but a tree is not highlighted. Detecting objects also aids an autonomous driving system in precise vehicle control and avoidance of potential collisions with other objects.

[0037] In some embodiments where loss calculator 160 receives the results of object detector 160, loss calculator 160 determines loss 165 based on a combination of object loss and a importance-based mean squared error loss algorithm. The object loss is the sum of the average of classification loss and regression loss for each detected object. Each of the classification loss and regression loss is a cross entropy loss. In some approaches, classification loss in object detection is determined using the following equation (3):

number

number

[0038] In some approaches, the regression loss is determined using the following equation (5):

number

number

[0039] In some embodiments, the importance of the detected object M i is calculated by determining a group of pixels in the reconstruction data 150 that contains a corresponding detected object. An importance determination is performed for each pixel using the group of pixels. In some embodiments, the importance determination is similar to the importance determination described above with respect to the additional information 110. The importance determination is then averaged over the group of pixels, i.e., the importance values ​​for each pixel are added and then divided by the number of pixels in the group of pixels.

[0040] In some embodiments, the loss 165 is determined using the following equation (7):

number

[0041] 2 is a flow diagram of a method 200 of compressing data, according to some embodiments. In some embodiments, method 200 is executable by data compression system 100. In operation 205, data is collected. In some embodiments, the data is collected from a single sensor. In some embodiments, the data is collected from multiple sensors. In some embodiments, each of the sensors is the same type of sensor. In some embodiments, at least one sensor is different from at least one other sensor. In some embodiments, the sensors include at least one of a LiDAR, a visible light sensor, an IR light sensor, a RADAR, a SONAR, or another suitable type of sensor. In some embodiments, the data collected in operation 205 corresponds to data 105 (FIG. 1).

[0042] In operation 210, a mask is generated, for example using a processor connected to the encoder 130 (FIG. 1). The mask is generated by pre-processing the data collected in operation 205 in operation 212 and applying rules to the pre-processed data in operation 214. In some embodiments, the applied rules are based on additional information 110 (FIG. 1). In some embodiments, the pre-processing includes a classification of the detected objects. In some embodiments, the pre-processing includes determining a speed of each detected object. In some embodiments, the pre-processing includes determining a distance of each detected object from a corresponding sensor. In some embodiments, the pre-processing is omitted. For example, in some embodiments where the sensor includes a LiDAR, pre-processing to determine the distance between the sensor and the detected object is not necessary because the distance between the sensor and each detected object has already been determined. Those skilled in the art will appreciate that in some embodiments, pre-processing to determine the object classification is performed even when the sensor is a LiDAR. In some embodiments, operation 210 generates the mask 120 (FIG. 1).

[0043] In operation 215, the data is encoded by an encoder, e.g., encoder 130 (FIG. 1), based on the mask generated in operation 210. In some embodiments, encoding the data includes reducing the resolution of portions of the data that have low importance values. Encoding the data produces compressed data, e.g., compressed data 137 (FIG. 1).

[0044] In operation 220, the compressed data is transmitted, for example using a transmitter. In some embodiments, the compressed data is transmitted over a wireless network. In some embodiments, the compressed data is transmitted using a wired network. In some embodiments, the network includes network 135 (FIG. 1).

[0045] In operation 225, the transmitted data is decoded by a decoder, e.g., decoder 140 (FIG. 1). Decoding the transmitted data produces reconstructed data, e.g., reconstructed data 150 (FIG. 1). The reconstructed data differs from the data collected in operation 205. In some embodiments, portions of the reconstructed data are at a lower resolution compared to corresponding portions of the data collected in operation 205.

[0046] In operation 230, the vehicle is controlled based on the reconstruction data or the reconstruction data is monitored remotely, for example using a processor coupled to encoder 130 or decoder 140 (FIG. 1). In some embodiments where the vehicle includes an autonomous driving system, the reconstruction data is used to control the vehicle to avoid detected objects. In some embodiments where remote monitoring is performed, the reconstruction data is provided to a remote monitoring device.

[0047] Compared to other approaches, data compression method 200 reduces the latency between collecting data in act 205 and generating reconstruction data in act 225. By reducing the resolution of the data collected in act 205 or excluding portions of that data in generating the reconstruction data in act 225, the latency is reduced without significantly losing relevant information in the reconstruction data. Reducing the latency while maintaining relevant information means that an autonomous driving system can respond better to detected objects and / or remote monitoring devices can identify hazards more quickly and accurately.

[0048] In optional operation 235, a ground truth mask is generated, for example, using a processor coupled to the encoder 130 (FIG. 1). The ground truth mask is generated based on known data to help improve the generation of the mask in operation 210 as part of a feedback process. In some embodiments, the ground truth mask corresponds to the GT mask 170 (FIG. 1). In some embodiments, operation 235 is omitted when the mask is generated based on rules instead of machine learning. In some embodiments, operation 235 is omitted to reduce processing load.

[0049] In optional operation 240, object detection is performed on the reconstruction data. In some embodiments, object detection is performed using object detector 180 (FIG. 1). Object detection identifies objects in the reconstruction data. In some embodiments, object detection modifies the reconstruction data viewable by the remote monitoring device to highlight the detected object. For example, in some embodiments, a box or other identification system is placed around or adjacent to the detected object to attract the attention of the remote monitoring device. In some embodiments, the highlighting is performed based on a classification of the detected object. For example, in some embodiments, a vehicle is highlighted but a tree is not highlighted. Detecting objects also helps the autonomous driving system to more precisely control the vehicle and avoid potential collisions with other objects. In some embodiments, operation 240 is omitted. In some embodiments, operation 240 is omitted to reduce processing load.

[0050] In operation 245, the loss is calculated based on a combination of regression loss, classification loss, and mean squared error loss, e.g., using a processor coupled to encoder 130 or decoder 140 (FIG. 1). In some embodiments, the loss is calculated using a importance-based mean squared error loss. In some embodiments, the loss is calculated based on a ground truth mask. In some embodiments, the loss is calculated based on object detection performed on the reconstruction data. In some embodiments, the loss corresponds to loss 165 (FIG. 1).

[0051] In operation 250, the mask, the encoder, and the decoder are updated based on the loss calculated in operation 245, e.g., using one or more processors coupled to the encoder 130 and / or the decoder 140 (FIG. 1). In some embodiments, only the encoder and the decoder are updated. The updates help improve the accuracy of the reconstruction data, improving the reliability of the autonomous driving system or the remote monitoring system.

[0052] In some embodiments, the order of operations in method 200 is adjusted. For example, in some embodiments, the ground truth mask in operation 235 is generated prior to the encoding of the data in operation 215, e.g., using a processor coupled to encoder 130 (FIG. 1). In some embodiments, additional operations are included in method 200. For example, in some embodiments, an alert is generated to be sent to a remote monitoring device. In some embodiments, the alert includes an audio alert, a text alert, or another suitable alert. In some embodiments, the alert is sent to a mobile device of the remote monitoring device. In some embodiments, operations of method 200 are removed. For example, in some embodiments, operation 212 is omitted if pre-processing of the data collected in operation 205 is not required.

[0053] 3A is a flow diagram of a method 300 for generating a mask, according to some embodiments. In some embodiments, the method 300 can be used to perform operation 210 of method 200 (FIG. 2). In some embodiments, the method 300 can be used to generate the mask 120 (FIG. 1). Data 305 is received from a sensor. In some embodiments, the data 305 is similar to data 105 (FIG. 1).

[0054] In operation 320, a mask is generated, for example using a processor coupled to encoder 130 (FIG. 1). In operation 325, pre-processing is performed on data 305, for example using a processor coupled to encoder 130 (FIG. 1). The additional information is provided to operation 320 so as to be usable by both operation 325 and operation 327. In some embodiments, pre-processing is performed without relying on additional information 310. In some embodiments, to reduce processing load, pre-processing is performed taking into account the additional information. In some embodiments, the additional information is similar to additional information 110 (FIG. 1). In some embodiments, pre-processing includes classifying the detected objects, determining the velocity of each detected object, determining the distance of each detected object from the corresponding sensor, or another suitable pre-processing operation.

[0055] In operation 327, rules from additional information 310 are applied to the preprocessed data to generate a mask 329. In some embodiments, mask 329 is similar to mask 120 (FIG. 1). In some embodiments, the rules include rules regarding the velocity of detected objects, rules regarding the distance of objects from a corresponding sensor, rules regarding classification of objects, or a combination of rules.

[0056] 3B is an image 330 received by a sensor, according to some embodiments. In some embodiments, image 330 is captured by a visible light sensor or an infrared light sensor. In some embodiments, image 330 can be used as data 105 (FIG. 1), data 305 (FIG. 3A), or data collected in operation 205 (FIG. 2).

[0057] FIG. 3C is a point cloud 340 received by a sensor, according to some embodiments. In some embodiments, the point cloud 340 is captured by a LiDAR. In some embodiments, the point cloud 340 can be used as data 105 (FIG. 1), data 305 (FIG. 3A), or data collected in operation 205 (FIG. 2). The point cloud 340 is captured in the same view as the image 330 (FIG. 3B). In comparison to the image 330, the point cloud 340 includes dots indicating positions where light from the LiDAR sensor is reflected back towards the sensor. A 3D rendering of the point cloud 340 shows the LiDAR's ability to determine the distance of objects from the sensor. In some embodiments, the point cloud 340 is converted into a two-dimensional (2D) representation, such as a range image or an overhead image.

[0058] 3D is a mask 350 that can be applied to sensor data, according to some embodiments. Mask 350 includes multiple regions 355. Each of the regions 355 represents a pixel grouping of data used to generate mask 350. Mask 350 is generated based on the same view as image 330 (FIG. 3B) and point cloud 340 (FIG. 3C). Mask 350 also includes an importance value for each of regions 355. The importance values ​​for regions 355 are generated based on a combination of rules, such as object distance and object classification.

[0059] If we cover the image 330 (FIG. 3B) with a mask, we can see that there is a train track along the right side of the image. The train track is less likely to obstruct vehicles along the road. Thus, the importance values ​​along the right side of the mask indicate low importance. The value of the bottom right region 355 is higher than the other right region 355 due to the close proximity between the bottom right region and the sensor. The importance value at the top of the mask 350 corresponds to the sky in the image 330. Sky is of very low importance to the control of a vehicle during autonomous driving or to a remote monitoring device. There is a road in the path ahead of the sensor. The road is of high importance for the train to follow, but there are no vehicles on the road in front of the sensor, so the importance value is less than the maximum value. Again, the importance value decreases along the road as the distance from the sensor increases. Along the left side of the mask 350, region 355 has a high importance value. As can be seen in the image 330, vehicles are parked along the side of the road. These vehicles are at higher collision risk than other objects in the image. Thus, regions 355 containing these vehicles are weighted more. Again, regions 355 closer to the sensor are weighted more as important as regions 355 further from the sensor that contain similar objects. In some embodiments, point cloud data within a minimum threshold distance of the sensor are given a lower importance value since there are fewer objects expected to be located within such a close distance. In some embodiments, the minimum threshold distance is less than 100 m. In some embodiments, the minimum threshold distance is less than 50 m. In some embodiments, the minimum threshold distance is less than 10 m.

[0060] 4 is a flow diagram of a method 400 of applying rules to received data, according to some embodiments. In some embodiments, method 400 can be used to perform operation 327 (FIG. 3A). In some embodiments, method 400 can be used to perform operation 314 (FIG. 2). In operation 405, an object is detected, for example using object detector 180 (FIG. 1). In some embodiments, the object is detected based on a database of object images. In some embodiments, the object is detected based on other parameters, such as heat signature, size, motion, or other suitable parameters.

[0061] In operation 410, the detected objects are classified, for example using object detector 180 (FIG. 1). In some embodiments, the classification of the detected objects is based on user input. In some embodiments, the classification of the detected objects is based on a database. In some embodiments, the classification of the objects is based on distance from a sensor.

[0062] In operation 415, one or more rules are applied to the detected objects based on the classification, for example using a processor coupled to encoder 130 or decoder 140 (FIG. 1). The rules are applied to determine the importance of each of the detected objects. In some embodiments, the rules are similar to additional information 110 (FIG. 1) described above. In some embodiments, the rules are applied based on proximity to the sensor, motion of the detected object, type of detected object, or a combination thereof. Following application of the rules to the detected objects, a mask is generated. In some embodiments, the generated mask is similar to mask 120 (FIG. 1).

[0063] FIG. 5 is a flow diagram of a method 500 for generating a ground truth mask according to some embodiments. In some embodiments, the method 500 can be used to generate the GT mask 170 (FIG. 1). The method 500 includes receiving data 505, GT segmentation rules 510, and additional information 515. In some embodiments, the GT segmentation rules 510 are similar to the GT segments 175 (FIG. 1). In some embodiments, the additional information 515 is similar to the additional information 177 (FIG. 1). In some embodiments, the data 505 is similar to the data 105 (FIG. 1). In operation 520, the GT mask is generated based on the data 505, the ground truth segments 510, and the additional information 515. Compared to generating a mask in, for example, the method 300 (FIG. 3), the method 500 includes the ground truth segments 510 based on known information. By including the known information along with the data 505 and the additional information, the ground truth mask 530 generates a result based on an error-free detection result. As a result, the ground truth mask has a higher confidence level and can be used to train a mask, such as mask 120 (FIG. 1), during data encoding. Operation 520 outputs ground truth mask 530.

[0064] FIG. 6 is a flow diagram of a method 600 of using a data compression system, according to some embodiments. In some embodiments, at least a portion of the method 600 is implemented using the data compression system 100 (FIG. 1). In some embodiments, the method 600 implements the remote monitoring aspect of the present disclosure. In remote monitoring, a remote monitoring device receives the data and determines whether any type of intervention is recommended by the remote monitoring device. The remote monitoring device may continue to monitor the impact of any intervention to determine whether to terminate or modify the intervention. In some embodiments, the remote monitoring device is a user. In some embodiments, the remote monitoring device is automated.

[0065] In operation 605, data is collected, for example, using sensors used to collect data 105 (FIG. 1). In some embodiments, operation 605 is similar to operation 205 (FIG. 2).

[0066] In operation 610, the data is encoded based on the mask, for example, using a processor coupled to encoder 130 (FIG. 1). In some embodiments, operation 610 is similar to a combination of operations 210 and 215 (FIG. 2).

[0067] The encoded data is transmitted, for example using a transmitter, in operation 615. In some embodiments, operation 615 is similar to operation 220 (FIG. 2).

[0068] In operation 620, the transmitted data is decoded, for example, using decoder 140 (FIG. 1). In some embodiments, operation 620 is similar to operation 225 (FIG. 2).

[0069] In optional operation 625, the reconstructed data is displayed using a display, such as, for example, a handheld device. The reconstructed data is obtained by decoding the transmitted data from operation 620. In some embodiments, the portion of the reconstructed data is at a lower resolution than the portion of the data collected in operation 605. In some embodiments, the reconstructed data includes highlighting of detected objects. In some embodiments, the reconstructed data does not include highlighting of detected objects. In some embodiments, the reconstructed data includes a visible light image. In some embodiments, the reconstructed data includes a point cloud image. In some embodiments, the reconstructed data is displayed on multiple displays. In some embodiments, the reconstructed data is displayed in response to a selection of the reconstructed data by the remote monitoring device. That is, the reconstructed data is available for display but remains undisplayed until the remote monitoring device selects the reconstructed data.

[0070] In some embodiments, the reconstruction data is displayed in response to an alert being generated, for example, using a processor coupled to decoder 140 (FIG. 1). For example, in some embodiments, an alert is generated and transmitted to a remote monitoring device in response to a vehicle colliding with an object. In some embodiments, the alert is an audio alert. In some embodiments, the alert is a text alert. In some embodiments, the alert is transmitted to a terminal, such as a mobile terminal, that can be viewed by the remote monitoring device. In response to the alert, the remote user may select the reconstruction data. By including the alert, the risk that the remote monitoring device will not be able to identify a potential problem is reduced.

[0071] In some embodiments, operation 625 is omitted. In some embodiments, operation 625 is omitted in response to the remote monitoring device being automated, i.e., the automated remote monitoring device does not need to display the reconfiguration data.

[0072] In operation 630, a determination is made, for example, by a user, such as a remote viewer, or using a processor connected to decoder 140 (FIG. 1), as to whether intervention is recommended. Intervention is an action that the remote monitoring device may take to help mitigate a problem identified by the reconstruction data. For example, in some embodiments where the reconstruction data indicates a collision by a vehicle, the remote monitoring device may send emergency assistance to the location of the vehicle. In some embodiments where the reconstruction data indicates a congestion, the remote monitoring device may provide recommended rerouting instructions to vehicles, including manned and / or autonomous vehicles.

[0073] A decision as to whether or not an intervention is recommended is made based on whether any of the intervention options available at the location of the collected data corresponding to the reconstruction data improve the situation indicated by the reconstruction data. If improvement is possible based on the available intervention options, an intervention is recommended. If improvement is not possible based on the available intervention options, an intervention is not recommended. One skilled in the art will note that not all intervention options are available in all locations. For example, traffic light control is more likely to be an option available in an urban area than in a rural area.

[0074] In response to a determination that intervention is not recommended, the method returns, as indicated by "No," to operation 605. In response to a determination that intervention is recommended, the method proceeds to operation 635, as indicated by "Yes."

[0075] In operation 635, a type of intervention is determined, for example, by a user, such as a remote viewer, or using a processor connected to the decoder 140 (FIG. 1). The type of intervention is an intervention that the remote monitoring device will implement based on the reconstruction data. The type of intervention is determined based on intervention options available at the location of the data collected from operation 605 that corresponds to the reconstruction data. In some embodiments, in response to a determination that an intervention is recommended, the intervention options are displayed on the remote monitoring device. In some embodiments, the type of intervention is selected based on which type of intervention will result in the greatest improvement to the situation represented by the reconstruction data. In some embodiments, the type of intervention is selected based on which type of intervention will most quickly affect the situation represented by the reconstruction data. For example, in some embodiments where the reconstruction data indicates a collision, changing traffic signals to route vehicles around the collision will have a more rapid impact than sending emergency assistance to the location of the collision. In some embodiments, the type of intervention is recommended based on a computer analysis of the reconstruction data.

[0076] In operation 640, the intervention signal is transmitted, for example by a transmitter. In some embodiments, the intervention signal is transmitted over a wireless network. In some embodiments, the intervention signal is transmitted over a wired network. In some embodiments, the network over which the intervention signal is transmitted is the same network over which the encoded data is transmitted. In some embodiments, the network over which the intervention signal is transmitted is different than the network over which the encoded data is transmitted. In some embodiments, the destination of the transmitted intervention signal is based on the type of intervention determined in operation 635. For example, in some embodiments where the type of invention is to send emergency assistance, the intervention signal is transmitted to police or other emergency responders. In some embodiments where the type of invention is to adjust a traffic signal, the intervention signal is transmitted to a traffic signal near the location of the collected data corresponding to the reconstruction data.

[0077] In some embodiments, a notification signal is sent to the vehicle indicating the type of intervention selected, hi some embodiments, the notification signal also includes an estimated time of arrival of the selected intervention.

[0078] In operation 645, a determination is made, for example by a user, such as a remote viewer, or using a processor connected to the decoder 140 (FIG. 1), as to whether the situation indicated by the reconstruction data is resolved. To make the determination, the remote monitoring device continues to monitor the reconstruction data over time. If the situation returns to the expected situation, the situation is determined to be resolved. In response to a determination that the situation has not been resolved, the method 600 returns to operation 635, where the remote monitoring device determines whether additional or different types of intervention should be implemented, indicated by "no". In response to a determination that the situation has been resolved, the method 600 proceeds to operation 650, indicated by "yes".

[0079] At operation 650, the intervention effected by the signal transmitted at operation 640 is removed, for example, by a user, such as a remote viewer, or using a processor connected to decoder 140 (FIG. 1). In some embodiments, the removal of the intervention includes transmitting a signal to return operation to a normal setting. In some embodiments, the removal of the intervention includes transmitting a signal indicating that no further emergency assistance is requested. In some embodiments, a notification signal is transmitted to the vehicle indicating that the intervention has ended. In some embodiments, the notification signal includes a request to confirm that no further intervention is requested by the driver or passengers in the vehicle. In response to a request for additional intervention, method 600 returns to operation 630.

[0080] Using method 600 that includes the use of a mask to encode collected data, the delay time between the occurrence of an event and the implementation of an intervention from the remote monitoring device is reduced without sacrificing the accuracy of the remote monitoring device's ability to identify the occurrence of the event. By reducing the amount of data transmitted due to the use of a mask, the processing load and delay time are reduced, which may shorten the response time for intervention. This has benefits to the driver or passengers of the vehicle that includes the sensor collecting the data, as well as to drivers and passengers of other vehicles that may be routed around the event that may cause congestion or danger to the drivers and passengers.

[0081] In some embodiments, the operations of method 600 are performed in a different order. For example, in some embodiments, operation 635 is performed before operation 630, whereby the determination of whether intervention is recommended is based on the type of intervention selectable by the remote monitoring device. In some embodiments, at least one operation of method 600 is omitted. For example, in some embodiments, operation 625 is omitted when the remote monitoring device is automated. In some embodiments, additional operations are included in method 600. For example, in some embodiments, a notification signal is sent to a vehicle that includes a sensor that collects data.

[0082] FIG. 7 is a schematic diagram of a system 700 for implementing a data compression system and / or a data compression method, according to some embodiments. In some embodiments, the system 700 can be used to implement at least a portion of the data compression system 100 (FIG. 1). In some embodiments, the system 700 can be used to implement at least a portion of the method 200 (FIG. 2), the method 300 (FIG. 3A), the method 400 (FIG. 4), the method 500 (FIG. 5), and / or the method 600 (FIG. 6). The system 700 includes a hardware processor 702 and a non-transitory computer-readable storage medium 704 encoded with, i.e., storing, a computer program code 706, i.e., a set of executable instructions. The computer-readable storage medium 704 is also encoded with instructions 707 for interfacing with an external device. The processor 702 is electrically connected to the computer-readable storage medium 704 via a bus 708. The processor 702 is also electrically connected to an input / output (I / O) interface 710 by the bus 708. The network interface 712 is also electrically connected to the processor 702 via the bus 708. The network interface 712 is connected to a network 714, such that the processor 702 and the computer readable storage medium 704 may connect to external elements via the network 714. The processor 702 is configured to execute computer program code 706 encoded on the computer readable storage medium 704 to enable the system 700 to perform some or all of the operations described in the data compression system 100, the method 200, the method 300, the method 400, the method 500, and / or the method 600.

[0083] In some embodiments, processor 702 is a central processing unit (CPU), a multiprocessor, a distributed processing system, an application specific integrated circuit (ASIC), and / or any other suitable processing device.

[0084] In some embodiments, computer readable storage medium 704 is an electronic, magnetic, optical, electromagnetic, infrared, and / or semiconductor system (or apparatus or device). For example, computer readable storage medium 704 includes a semiconductor or solid state memory, a magnetic tape, a removable computer diskette, a random access memory (RAM), a read only memory (ROM), a rigid magnetic disk, and / or an optical disk. In some embodiments using an optical disk, computer readable storage medium 504 includes a compact disk read only memory (CD-ROM), a compact disk read / write (CD-R / W), and / or a digital video disk (DVD).

[0085] In some embodiments, the storage medium 704 stores computer program code 706 configured to cause the system 700 to perform some or all of the operations described in the data compression system 100, method 200, method 300, method 400, method 500, and / or method 600. In some embodiments, the storage medium 704 also stores information necessary to perform some or all of the operations described in the data compression system 100, method 200, method 300, method 400, method 500, and / or method 600, as well as information generated during the performance of some or all of the operations described in the data compression system 100, method 200, method 300, method 400, method 500, and / or method 600, such as, for example, rule parameters 716, mask parameters 718, ground truth mask parameters 720, object library parameters 722, intervention option parameters 724, and / or sets of executable instructions for performing some or all of the operations described in the data compression system 100, method 200, method 300, method 400, method 500, and / or method 600.

[0086] In some embodiments, storage medium 704 stores instructions 707 for interfacing with an external device. The instructions 707 may cause processor 702 to generate instructions readable by an external device to effect some or all of the operations described in data compression system 100, method 200, method 300, method 400, method 500, and / or method 600.

[0087] System 700 includes an I / O interface 710. I / O interface 710 is connected to external circuitry. In some embodiments, I / O interface 710 includes a keyboard, keypad, mouse, trackball, trackpad, and / or cursor direction keys for communicating information and commands to processor 702.

[0088] The system 700 also includes a network interface 712 coupled to the processor 702. The network interface 712 allows the system 700 to communicate with a network 714 to which one or more other computer systems are connected. The network interface 712 may include a wireless network interface, such as BLUETOOTH, WIFI, WIMAX, General Packet Radio Service (GPRS), or Wideband Code Division Multiple Access (WCDMA), or a wired network interface, such as ETHERNET, Universal Serial Bus (USB), or IEEE-1394 (Institute of Electrical and Electronics Engineers 1394). In some embodiments, some or all of the operations described in the data compression system 100, method 200, method 300, method 400, method 500, and / or method 600 are performed in two or more systems 700, and information is exchanged between the various systems 700 via the network 714.

[0089] (Additional Note)

[0090] One aspect of the present disclosure relates to a system. The system comprises a non-transitory computer-readable medium configured to store instructions, and a processor coupled to the non-transitory computer-readable medium. The processor is configured to execute instructions for generating a mask based on received data from a sensor, the mask including a plurality of importance values, each region of the received data being assigned to a corresponding importance value of the plurality of importance values. The processor is further configured to execute instructions for encoding the received data based on the mask. The processor is further configured to execute instructions for transmitting the encoded data to a decoder to define reconstruction data. The processor is further configured to execute instructions for calculating a loss based on the reconstruction data, the received data, and the mask. The processor is further configured to execute instructions for providing training to an encoder for encoding the received data based on the calculated loss.

[0091] In some embodiments, the processor is further configured to execute instructions for decoding the received data to define the reconstruction data.

[0092] In some embodiments, the processor is further configured to execute instructions for generating a mask based on the proximity of the detected object to the sensor.

[0093] In some embodiments, the processor is further configured to execute instructions for generating a mask based on the detected movement of the detected object based on the received data.

[0094] In some embodiments, the processor is further configured to execute instructions for generating a mask based on a classification of each of the objects detected in the received data.

[0095] In some embodiments, the processor is further configured to execute instructions for generating a ground truth mask based on the received data and the ground truth segments.

[0096] In some embodiments, the processor is further configured to execute instructions for calculating the loss further based on the ground truth mask.

[0097] In some embodiments, the processor is further configured to execute instructions for detecting objects in the reconstruction data.

[0098] In some embodiments, the processor is further configured to execute instructions for calculating the loss further based on objects detected in the reconstruction data.

[0099] One aspect of the present disclosure relates to a method. The method includes using a processor to generate a mask based on received data from a sensor, the mask including a plurality of importance values, each region of the received data being assigned to a corresponding importance value of the plurality of importance values. The method further includes encoding the received data based on the mask. The method further includes transmitting the encoded data to a decoder to define reconstructed data. The method further includes calculating a loss based on the reconstructed data, the received data, and the mask. The method further includes providing training to an encoder for encoding the received data based on the calculated loss.

[0100] In some embodiments, the method further includes decoding the received data to define the reconstruction data.

[0101] In some embodiments, generating the mask includes generating the mask based on a proximity of the detected object to the sensor.

[0102] In some embodiments, generating the mask includes generating the mask based on detected movement of the detected object based on the received data.

[0103] In some embodiments, generating the mask includes generating the mask based on a classification of each of the objects detected in the received data.

[0104] In some embodiments, the method further includes generating a ground truth mask based on the received data and the ground truth segments.

[0105] In some embodiments, computing the loss further comprises computing the loss based on a ground truth mask.

[0106] In some embodiments, the method further includes detecting the object in the reconstruction data.

[0107] In some embodiments, calculating the loss further includes calculating the loss based on detected objects in the reconstruction data.

[0108] One aspect of the present disclosure relates to a system. The system includes a sensor configured to collect data. The system further includes a non-transitory computer-readable medium configured to store instructions and rules. The system further includes a processor connected to the non-transitory computer-readable medium. The processor is configured to execute instructions for generating a mask based on the stored rules and the collected data. The system further includes an encoder configured to encode the collected data based on the mask. The system further includes a decoder configured to decode information received from the encoder to define reconstruction data. The processor is further configured to execute instructions for calculating a loss based on the reconstruction data, the collected data, and the mask, and updating at least one of the encoder or the decoder based on the calculated loss.

[0109] In some embodiments, the sensor includes a Light Detection and Ranging (LiDAR) sensor.

[0110] One aspect of the present disclosure relates to a system. The system includes a sensor configured to collect data. The system further includes a non-transitory computer-readable medium configured to store instructions and rules. The system further includes a processor connected to the non-transitory computer-readable medium. The processor is configured to execute instructions for generating a mask based on the stored rules and the collected data. The system further includes an encoder configured to encode the collected data based on the mask. The system further includes a decoder configured to decode information received from the encoder to define reconstruction data. The processor is further configured to execute instructions for calculating a loss based on the reconstruction data, the collected data, and the mask, and updating at least one of the encoder or the decoder based on the calculated loss.

[0111] In some embodiments, the sensor includes a Light Detection and Ranging (LiDAR) sensor.

[0112] The above outlines features of some embodiments so that those skilled in the art may conveniently understand aspects of the present disclosure. Those skilled in the art should appreciate that they may readily use the present disclosure as a basis for designing or modifying other processes and structures to carry out the same purposes and / or achieve the same advantages as the embodiments introduced herein. Those skilled in the art should also appreciate that such equivalent constructions do not depart from the spirit and scope of the present disclosure, and that various changes, substitutions, and alterations may be made herein without departing from the spirit and scope of the present disclosure.

[0113] In some embodiments, the processor comprises a plurality of processors, a first processor of the plurality of processors being at an encoder position and a second processor of the plurality of processors being at a decoder position different from the encoder position.

[0114] This application claims the benefit of U.S. Patent Application No. 17 / 467,282, filed September 6, 2021, which is incorporated by reference herein in its entirety.

Claims

1. a non-transitory computer-readable medium configured to store instructions; a processor coupled to the non-transitory computer readable medium, the processor comprising: generating a mask based on received data from a sensor, the mask including a plurality of importance values, each region of the received data being assigned a corresponding importance value of the plurality of importance values; encoding the received data based on the mask; transmitting the encoded data to a decoder to define reconstruction data; calculating a loss based on the reconstructed data, the received data, and the mask; and providing training to an encoder for encoding the received data based on the calculated loss; configured to execute the instructions for system.

2. The processor, Decoding the received data to define the reconstruction data. The system of claim 1 , further configured to execute the instructions for:

3. The processor, Generating the mask based on the proximity of detected objects to the sensor The system of claim 1 , further configured to execute the instructions for:

4. The processor, generating the mask based on the detected movement of the detected object based on the received data; The system of claim 1 , further configured to execute the instructions for:

5. The processor, generating the mask based on a classification of each of the objects detected in the received data; The system of claim 1 , further configured to execute the instructions for:

6. The processor, Generating a ground truth mask based on the received data and the ground truth segments. The system of claim 1 , further configured to execute the instructions for:

7. The processor, Calculate the loss further based on the ground truth mask. The system of claim 6 , further configured to execute the instructions for:

8. A processor comprising: generating a mask based on received data from a sensor, the mask including a plurality of importance values, each region of the received data being assigned a corresponding importance value of the plurality of importance values; encoding the received data based on the mask; and transmitting the encoded data to a decoder to define reconstruction data; calculating a loss based on the reconstructed data, the received data, and the mask; providing training to an encoder for encoding the received data based on the calculated loss; A method comprising:

Images