Vehicle data collection system and method of use thereof

Abstract

A vehicle data collection system and a method of using the same are provided. The vehicle data collection system includes an in-vehicle sensor, a non-transitory computer readable medium configured to store commands, and a processor connected to the non-transitory computer readable medium. The processor is configured to execute commands for generating an invariant feature map using a first neural network and comparing the invariant feature map with template data received from a server to determine a degree of similarity between the invariant feature map and the template data. The processor is further configured to execute commands for determining whether the determined degree of similarity exceeds a predetermined threshold and, in response to a determination that the determined degree of similarity exceeds the predetermined threshold, instructing a transmission unit to transmit sensor data to the server.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to a vehicle data collection system and a method for using the same. BACKGROUND

[0002] A vehicle is equipped with a wide variety of sensors. The sensors include cameras, light detection and ranging (LiDAR) sensors, and other types of sensors. These sensors are capable of collecting data from the environment surrounding the vehicle. A portion of the collected data is stored within the vehicle for use by the vehicle or for later retrieval by an external device.

[0003] A data collection system based on information obtained from vehicle sensors is based on launch conditions. These launch conditions are conditions of a pre-defined rule base. These launch conditions are stored in a memory within the vehicle and, in response to satisfaction of the launch conditions, pre-set data is collected from the sensors within the vehicle. Examples of launch conditions include hard braking, hard acceleration, hard rotation of the steering wheel, entry of the vehicle into a particular zone, and malfunction of the vehicle. SUMMARY

[0004] One aspect of the specification relates to a vehicle data collection system. The vehicle data collection system includes an on-board sensor. The vehicle data collection system also includes a non-transitory computer-readable medium configured to store commands. The vehicle data collection system includes a processor coupled to the non-transitory computer-readable medium. The processor is configured to execute the commands to generate an invariant feature map using a first neural network. The processor is also configured to execute the commands to compare the invariant feature map to template data received from a server and determine a similarity between the invariant feature map and the template data. The processor is also configured to execute the commands to determine whether the determined similarity exceeds a predetermined threshold. The processor is also configured to execute the commands to instruct a communication component to transmit sensor data to the server in response to a determination that the determined similarity exceeds the predetermined threshold. In some embodiments, the processor is also configured to execute the commands to detect an object within sensor data acquired by the on-board sensor and generate the invariant feature map based on the detected object. In some embodiments, the on-board sensor includes at least one of a camera, a light detection and ranging (LiDAR) sensor, and a radio detection and ranging (RADAR) sensor. In some embodiments, the template data includes a target invariant feature map generated based on a target. In some embodiments, the processor is also configured to execute the commands to remove confidential information from the sensor data prior to transmitting the sensor data to the server. In some embodiments, the template data includes weights for the neural network, and the processor is configured to execute the commands to generate the invariant feature map using the first neural network with the weights. In some embodiments, the template data includes weights for the neural network, and the processor is also configured to execute the commands to determine the similarity using a second neural network with the weights.

[0005] One aspect of the present disclosure relates to a vehicle data collection system. The vehicle data collection system includes a graphical user interface (GUI) configured to receive target information from a user. The vehicle data collection system also includes a non-transitory computer readable medium configured to store commands. The vehicle data collection system also includes a processor coupled to the non-transitory computer readable medium. The processor is configured to execute the commands to receive the target information from the GUI. The processor is also configured to execute the commands to generate a target-invariant feature map using a first neural network based on the target information. The processor is also configured to execute the commands to generate a data collection instruction based on the target-invariant feature map. The processor is also configured to execute the commands to instruct a transmission unit to transmit the data collection instruction to at least one vehicle. The processor is also configured to execute the commands to receive sensor data from the at least one vehicle in response to the data collection instruction. The processor is also configured to execute the commands to report the sensor data to the user. In some embodiments, the target information includes an image. In some embodiments, the processor is configured to execute the commands to instruct the transmission unit to transmit the data collection instruction to a plurality of vehicles. In some embodiments, the processor is configured to execute the commands to generate the data collection instruction including a priority of a data requirement from the vehicle. In some embodiments, the processor is configured to execute the commands to generate the data collection instruction including a command for preventing the vehicle from removing confidential information from the sensor data based on an identity of the user. In some embodiments, the processor is configured to execute the commands to report the sensor data to the user with at least one of time and location information associated with the sensor data. In some embodiments, the processor is configured to execute the commands to report the sensor data to the user by generating an alert on a mobile device operated by the user.

[0006] At least one aspect of the specification relates to a method of using a vehicle data collection system. The method includes a step of collecting sensor data using at least one on-board sensor. The method also includes a step of generating an invariant feature map using a first neural network. The method further includes a step of comparing the invariant feature map to template data received from a server and determining a degree of similarity between the invariant feature map and the template data. The method includes a step of determining whether the determined degree of similarity exceeds a predetermined threshold. The method also includes a step of transmitting the sensor data to the server in response to a determination that the determined degree of similarity exceeds the predetermined threshold. In some embodiments, the method further includes a step of detecting an object within the sensor data, where the step of generating the invariant feature map is based on the detected object. In some embodiments, the on-board sensor includes at least one of a camera, a light detection and ranging (LiDAR) sensor, and a radio detection and ranging (RADAR) sensor. In some embodiments, the template data includes a target invariant feature map generated based on a target. In some embodiments, the method further includes a step of removing confidential information from the sensor data prior to transmitting the sensor data to the server. In some embodiments, the template data includes weights for the neural network, and the step of generating the invariant feature map includes a step of using the weights within the first neural network. BRIEF DESCRIPTION OF DRAWINGS

[0007] Aspects of the disclosure are described below with reference to the accompanying drawings. Methods and systems are provided for collecting data from a vehicle. In some embodiments, a vehicle data collection system includes a first neural network configured to generate an invariant feature map based on sensor data collected by an on-board sensor of the vehicle. The vehicle data collection system also includes a second neural network configured to determine a degree of similarity between the invariant feature map and template data received from a server. The vehicle data collection system further includes a processor configured to determine whether the determined degree of similarity exceeds a predetermined threshold. The vehicle data collection system also includes a transmitter configured to transmit the sensor data to the server in response to a determination that the determined degree of similarity exceeds the predetermined threshold. Figure One The disclosure can best be understood by reading the following detailed description provided in conjunction with the accompanying drawings. Notably, the various features are not drawn to scale in order to emphasize certain features of the disclosure. The dimensions of the various features can be arbitrarily increased or decreased for the sake of discussion.

[0008] Figure 1 is a schematic diagram of a vehicle data collection system of some embodiments.

[0009] Figure 2 is a flowchart of a method of generating an invariant feature map of some embodiments.

[0010] Figure 3 is a flowchart of a method of determining a degree of similarity of some embodiments.

[0011] Figure 4 is a flowchart of a method of using a vehicle data collection system of some embodiments.

[0012] Figure 5 is a block diagram of a system for implementing a vehicle data collection system of some embodiments. DETAILED DESCRIPTION

[0013] The following disclosure provides a plurality of different embodiments or examples for implementing different features of the provided subject matter. For simplicity of the present disclosure, below, specific examples of components, values, actions, materials, arrangements, etc. are described. These are of course merely simple examples and are not intended to be limiting. Other components, values, actions, materials, arrangements, etc. are contemplated. For example, in the formation of the 2nd feature as a whole or on the 1st feature in the following description, embodiments can be included in which the 1st and 2nd features are formed in direct contact, and likewise, embodiments can also be included in which an additional feature can be formed between the 1st and 2nd features so that the 1st and 2nd features do not come into direct contact. Also, in the present disclosure, sometimes the same reference numerals and / or words are repeatedly used in various embodiments. The purpose of this repetition is simplicity and clarity, and it is not intended to determine the relationship between the various embodiments and / or components being discussed.

[0014] Also, in the present specification, for the purpose of describing the relationship of one element or feature illustrated in the drawing with respect to another element or feature and for easy description, spatial relative terms such as "under", "lower", "bottom", "upper", "top", etc. can be used. The spatial relative terms, in addition to the orientation (direction) described in the drawing, are intended to include different orientations of the device in use or in action. The device can be oriented in other forms (rotated by 90 degrees or in other orientations), and the spatial relative terms used in the present specification can also be interpreted accordingly.

[0015] Throughout the present specification, a vehicle is mentioned in most cases. It is considered that those skilled in the art will recognize that the mention of a vehicle is merely for simplicity of the description, and the present disclosure is not limited to the collection of data using a single vehicle. Of course, it is considered that those skilled in the art will recognize that the present disclosure can be applied to the collection of data by a single vehicle, a fleet of vehicles commonly owned, a group of vehicles owned by an individual, or other groups of vehicles based on the data explored.

[0016] Due to the dependence on pre-defined rule bases for the triggering conditions, the ability to maximize the usefulness of the data collected by the sensors within the vehicle is limited. In some cases, the desired triggering conditions are difficult to describe using the syntax of the rule base. In some cases, new criteria for triggering conditions are generated. For example, it is considered that in the case where it is known that a criminal is driving a certain type of vehicle, the use of the vehicle sensors of other vehicles in order to identify the places where the type of vehicle exists, contributes to the likelihood of the police arresting the criminal. It cannot be considered that the features of the type of vehicle and / or the color of the vehicle and / or the stains of the vehicle are pre-defined within the interior of the vehicle. Therefore, the ability to receive information related to new triggering conditions and to provide data related to these new triggering conditions, increases the usefulness of the data collected by the vehicle sensors.

[0017] Without the ability to update the triggering conditions, the risk of false negatives or false positives in the collection of data increases. For example, without the ability to provide updated triggering conditions or the ability to describe the triggering conditions in a form that is difficult to use the syntax, as a result, even if the triggering conditions actually occur, the individual or multiple vehicles are unable to report the detection of the triggering conditions, i.e., false negatives. Or, in the case where the individual or multiple vehicles provide a large amount of data due to overly broad triggering conditions, the risk of the truly relevant data being buried within the large amount of data is higher. As a result, there is a risk that the truly relevant data is overlooked and not considered, i.e., false positives.

[0018] For the purpose of reducing false positives and false negatives, to develop an invariant feature map based on the data provided by the user, a neural network can be used. And, the invariant feature map is sent to the individual or multiple vehicles, which is used to develop a template for comparison with the data collected by the vehicle sensors. The individual or multiple vehicles identify data having a high degree of similarity to the received invariant feature map, which is transmitted to the server. Then, the server collects the data, which is provided to the user. By using the invariant feature map as the basis for comparison, the dependence on the syntax library of the triggering conditions is reduced. Also, data related to new triggering conditions can be collected. Also, the data comparison is more accurate, which is believed to result in fewer false positives and / or false negatives in the data collected. This system also, in some instances, can reduce the amount of data transmitted from the vehicle to the server, which reduces the processing load in both the vehicle and the server and the power used to transmit the data. This reduction in the processing load in the vehicle and the amount of transmission power consumed helps improve the fuel economy of the internal combustion vehicle and / or reduces the battery charge consumption in a hybrid vehicle or an electric vehicle.

[0019] Figure 1 FIG. 1 is a schematic diagram of a vehicle data collection system 100 according to some embodiments. The vehicle data collection system 100 includes a vehicle system VS and a server 18. The vehicle system VS is configured to communicate with the server 18. In some embodiments, the communication is wireless communication. In some embodiments, the communication is wired communication. The vehicle system VS is incorporated into a vehicle. In some embodiments, the vehicle is an automobile, such as a self-driving automobile, a motorcycle, a golf cart, or another suitable vehicle. The server 18 is configured to send data collection instructions to the vehicle system VS. The vehicle system VS receives the data collection instructions and sends data back to the server 18 based on the received data collection instructions.

[0020] The data collection instruction includes information related to an object or an occurrence as a target of the requested data. In some embodiments, the data collection instruction further includes information related to a period before and after the detection of the object or the occurrence as a target of the requested collected data. In some embodiments, the data collection instruction further includes information related to a type of the requested collected data, such as image data, location data, proximity data, and the like. The proximity data is information indicating how close the detected object is to the vehicle. In some embodiments, the data collection instruction further includes priority information for causing the priority of the response to the data collection instruction to be raised or lowered.

[0021] The vehicle system VS includes a camera 5 and other sensors 6 configured to acquire data of an environment surrounding the vehicle. The vehicle system further includes an electronic control unit (ECU) 4. The ECU 4 is configured to receive acquired images from the camera 5 and other collected data from the other sensors 6. The ECU 4 is configured to process the acquired images and the other collected data collectively referred to as collected data. In some embodiments, the camera 5 includes a visible light camera. In some embodiments, the camera 5 includes an infrared (IR) camera. In some embodiments, the camera 5 includes a combination of a visible light camera and an IR camera. In some embodiments, the camera 5 includes a depth sensing camera. In some embodiments, the other sensors 6 include a light detection and ranging (LiDAR) sensor, a radio wave sensor, a radio detection and ranging (RADAR) sensor, a sonar sensor, a GPS sensor, a speed sensor, a microphone, a fuel sensor, an oxygen sensor, a tire air pressure sensor, a thermometer, other suitable sensors, or a combination thereof. In some embodiments, the collected data includes image data, point cloud data, or other suitable data. The ECU 4 processes the collected data in order to determine which data should be provided to the server 18 in response to the data collection instruction in the presence of the data.

[0022] The ECU 4 is configured to implement various modules in order to process the collected data. The ECU 4 includes an object detection section 1 for detecting a type and a position of an object based on the collected data. The object detection section 1 includes a neural network (NN) 9 for object detection. The NN 9 for object detection is a trained NN for recognizing sensor data in order to determine the presence of an object within the sensor data. The NN 9 for object detection is configured to classify an object in order to determine which type of object is detected, such as a car, a bicycle, a pedestrian, a building, a traffic sign, and the like. The object detection section 1 outputs an object detection result 21 indicating a type and a position of each detected object.

[0023] The ECU 4 also includes a feature extraction section 2 configured to generate an invariant feature map with respect to each object detected based on the object recognition result 21 and information from the camera 5. The invariant feature map is a data map with respect to an image or a portion of an image that can be used to recognize a particular object. The invariant feature map includes abstracted information in which feature information related to the object and / or scene within the image is abstracted and information related to the characteristics of the vehicle system VS and the hardware (HW) configuration is removed. Within the invariant feature map, noise data is also reduced. That is, the distinct features of the object are quantified and recognized as abstracted information based on the image, such that the NN can recognize the object from other images regardless of the HW that acquired the image. In some embodiments, the feature extraction section 2 also receives information from other sensors 6. The feature extraction section 2 includes a feature extraction NN 14. The feature extraction NN 14 is a trained NN for extracting feature quantities of a detected object from image data that can be used to recognize a particular object and the object recognition result 21. In some embodiments, the feature extraction NN 14 is generic to vehicles including the ECU 4. In some embodiments, the feature extraction NN 14 is tuned (adjusted) for a particular vehicle including the ECU 4. For example, in some embodiments, the feature extraction NN 14 is configured to use historical record data related to previously processed collected data in order to assist in the determination of the invariant feature map. The feature extraction section 4 is configured to output the invariant feature map. In some embodiments, the feature extraction section 2 does not use the object recognition result 21 and generates the invariant feature map from data from the camera 5. In some embodiments, the feature extraction section 2 does not use camera data and generates the invariant feature map from data from other sensors 6, such as LiDAR. For example, the feature extraction section 2 generates the invariant feature map from point cloud data from a LiDAR sensor.

[0024] The ECU 4 also includes a component for determining a data collection event (data collection event determination section) 15. The data collection event determination section 15 is configured to determine the degree of similarity between the invariant feature map from the feature quantity extraction section 2 and the template data from the template storage section 17. The data collection event determination section 15 includes a similarity estimation NN 30. The similarity estimation NN 30 is a trained NN that determines the degree to which two items are similar to each other by comparing an invariant feature map with template data. In some embodiments, the similarity estimation NN 30 is trained using a triplet loss as a loss function. In some embodiments, the similarity estimation NN 30 has a SiameseNet (registered trademark) configuration. In some embodiments in which the data collection instruction includes information related to the weights used to generate the target invariant feature map, the similarity estimation NN 30 is configured to use the weights in determining the degree of similarity between the invariant feature map based on the camera 5 and the sensor 6 and the target within the template storage section 17.

[0025] The data collection event determination section 15 outputs the degree of similarity between the invariant feature map and the template data. In response to a determination that the degree of similarity between the invariant feature map and the template data exceeds a predetermined threshold value, the data collection event determination section 15 determines that the collection data should be transmitted to the server 18. The data collection event determination section 15 identifies the collection data that should be transmitted to the server 18 and sends a command to the data storage section 8 to write the collection data that should be transmitted to the storage unit 10. In response to a determination that the degree of similarity between the invariant feature map and the template data is below the predetermined threshold value, the data collection event determination section 15 determines that the collection data should not be transmitted to the server 18 and does not transmit a command to the storage unit 10 to prevent the collection data from being overwritten. In some embodiments, the protocol associated with the transmitted data includes storing the collection data until the collection data is sent. In some embodiments, the protocol associated with the transmitted data includes storing the collection data until a reception confirmation of the collection data is received from the server 18. In some embodiments, the protocol associated with the collection data includes storing the collection data until a command to delete the collection data is received. In some embodiments, the general protocol includes storing the collection data until the available storage space within the storage unit 10 falls below a storage threshold value. In some embodiments, the general protocol includes storing the collection data for a predetermined length of time.

[0026] In some embodiments, the data collection event determination section 15 is configured to determine whether the collected data includes private information or personal information (PI) or other confidential information. In some embodiments, the confidential information includes a person's face, a person's identifying information, or other secret data. In response to a determination that the collected data includes confidential information, the data collection event determination section 15 is configured to filter or remove the confidential information in some embodiments. In some embodiments, the data collection instruction includes authentication information or override information for enabling transmission of the confidential information. For example, in response to a data collection instruction from the police for looking for a certain person, collected data including a face of a person who is considered to have a similarity degree to the search target person that exceeds a threshold value can be transmitted. In some embodiments, to reduce the risk of inadvertently providing confidential information to a user who should not be allowed to receive the confidential information, the ability to override the filtering of the confidential information is limited to a certain level of users. In some embodiments, the certain level of users includes law enforcement agencies or other eligible users. In some embodiments, the certain level of users can be adjusted. In some embodiments, the data collection event determination section 15 further includes a second NN for filtering the confidential information. In some embodiments, the confidential information is filtered using the similarity degree estimation NN 30. In some embodiments, the NN 30 is trained such that the NN 30 can filter the confidential information by default, rather than an algorithm using a rule base determining whether the collected data includes the confidential information by the data collection event determination section 15.

[0027] ECU 4 also includes a receiving portion (receiving portion) 16 of data collection instructions. Receiving portion 16 is configured to receive template data and data collection instructions from server 18. In some embodiments, receiving portion 16 is a transceiving portion having the capability of both receiving and transmitting data. In some embodiments, receiving portion 16 is configured to receive data collection instructions and template data wirelessly. In some embodiments, receiving portion 16 is configured to receive data collection instructions and template data via a wired connection. In some embodiments, receiving portion 16 is configured to perform authentication with respect to received data collection instructions and / or template data prior to storing the data within ECU 4. In some embodiments, receiving portion 16 is configured to receive weight data that is to overwrite existing weight data used by neural network 30 for similarity estimation within VS. Receiving portion 16 is configured to send weight information to similarity estimation NN 30 for use by NN 30 in estimating similarity using weight data. In some embodiments, receiving portion 16 is configured to receive weight data that is to overwrite existing weight data used by neural network for similarity estimation 30 within VS for use in generating template data. Receiving portion 16 is configured to send template data to template storage portion 17. In some embodiments, receiving portion 16 is configured to send data collection instructions to storage unit 10.

[0028] ECU 4 also includes a template storage portion 17 configured to store template data received from receiving portion 16. Template storage portion 17 is configured to provide template data to data collection event determination portion 15 for comparison with invariance feature map. In some embodiments, template storage portion 17 is configured to store multiple templates. In some embodiments, template storage portion 17 includes a solid state memory or another type of memory device. In some embodiments, template storage portion 17 is integrated with storage unit 10. In some embodiments, template storage portion 17 is configured to store a template until a command is received to retrieve the stored template. In some embodiments, template storage portion 17 is configured to delete a stored template after a predetermined period of time has elapsed from initial storage of the template. In some embodiments, individual templates within template storage portion 17 have different protocols for deletion of each template. In some embodiments, the storage protocol for at least one template within template storage portion 17 is based on information received within a data collection instruction. In some embodiments, the storage protocol for at least one template within template storage portion 17 is based on a command from a user of the vehicle. In some embodiments, a template includes a target invariance feature map received from server 18.

[0029] The ECU 4 also includes a data transmission section (transmission section) 11. The transmission section 11 is configured to transmit the collected data based on the data collection instruction in response to a command from the data collection event determination section 15. In some embodiments, the transmission section 11 takes into account the priority information of the data collection instruction in deciding the timing for transmitting the collected data. In some embodiments, the transmission section 11 transmits the collected data in a certain period before and after the detection of the object or event for which the data collection instruction is satisfied based on the required period in the data collection instruction. In some embodiments, the transmission section 11 decides which type of data should be transmitted based on the data collection instruction. In some embodiments, the transmission section 11 transmits the invariant feature map. In the above description, the transmission section 11 includes deciding when which information should be transmitted, but in some embodiments, the decision of when which information should be transmitted in response to the data collection instruction is made by another constituent element other than the ECU 4, such as the data collection event determination section 15. In some embodiments, the transmission section 11 is integrated with the reception section 16 as part of a transceiver section. In some embodiments, the transmission section 11 is configured to transmit the collected data in a wireless manner. In some embodiments, the transmission section 11 is configured to transmit the collected data via a wired connection.

[0030] The ECU 8 also includes a data storage section 8 configured to store the collected data from the camera 5 and other sensors 6. In some embodiments, the data storage section 8 has a ring buffer configuration. In some embodiments, the data storage section 8 includes a solid state memory or another suitable memory device. In some embodiments, the data storage section 8 is configured to provide temporary storage for the collected data that is currently being processed by the ECU 4. In some embodiments, the data storage section 8 is configured to transmit the data to the storage unit 10 in response to the decision made by the data collection event determination section 15 to transmit the collected data to the server 18. In some embodiments, the data storage section 8 is configured to delete the collected data in response to the decision made by the data collection event determination section 15 not to transmit the collected data to the server 18. In some embodiments, the data storage section 8 is integrated with the storage unit 10. In some embodiments, the data storage section 8 is integrated with the template storage section 17.

[0031] The ECU 4 also includes a storage unit 10. The storage unit 10 is configured to store the collected data based on a command from the data collection event determination section 15. In some embodiments, the storage unit 10 is integrated with the data storage section 8 or the template storage section 17. In some embodiments, the storage unit 10 includes a solid state memory or other suitable memory. In some embodiments, the storage unit 10 is configured to store information related to the achievement of the data collection instruction. In some embodiments, the storage unit 10 is configured to store the collected data in association with the corresponding data collection instruction. In some embodiments, the storage unit 10 is configured to store collected data that is deemed unsuitable for transmission until the collected data is retrieved by a user.

[0032] The server 18 includes a graphical user interface (GUI) 24 configured to receive input from a user related to which target is being explored. In some embodiments, the GUI 24 includes a library of common targets for wildlife, weather events, points of interest, or other suitable targets. In some embodiments, the user is able to upload images or portions of images to the GUI 24. In some embodiments, the GUI 24 includes a restricted portion that is only accessible by a user that is authenticated by one who is a selected group of users. In some embodiments, the GUI 24 is configured to receive information related to a certain period of time before and after the target is detected in a manner that is included in the collected data. In some embodiments, the GUI 24 is configured to receive information related to a user-based priority requirement for receiving the collected data. In some embodiments, the GUI 24 is configured to receive information related to time, weather conditions, or other information related to images uploaded using the GUI 24.

[0033] The server 18 also includes a tracking target selection section 19 configured to receive target information from the GUI 24. The tracking target selection section 19 decides which requested target from the GUI 24 should be issued a data collection instruction. In some embodiments, the tracking target selection section 19 identifies the next requested target that should be the subject of a data collection instruction based on the inputted priority of the requested targets in the GUI 24. In some embodiments, the tracking target selection section 19 identifies the next requested target that should be the subject of a data collection instruction based on the identity of the user that inputted the target into the GUI 24. In some embodiments, the tracking target selection section 19 decides the next requested target that should be the subject of a data collection instruction based on a queue (first-in, first-out) or stack (first-in, last-out) process. In some embodiments, a combination of criteria is used by the tracking target selection section 19 in order to decide which requested target is the subject of a data collection instruction. In some embodiments, the tracking target selection section 19 automatically identifies the next requested target that should be the subject of a data collection instruction based on data in the collected data storage section 25 and / or a history of previous tracking target selections without a request from a user via the GUI 24.

[0034] The server 18 also includes a feature amount extraction NN 20 for generating a target invariant feature amount map corresponding to a requested target received from the GUI 24. The feature amount extraction NN 20 is configured to generate a hardware, independent, or less sensitive feature amount map of the sensors inside the vehicle that receive the data collection instruction. By generating a target invariant feature amount map that is independent of the hardware of the sensors inside the vehicle, the feature amount extraction NN 20 is able to create a target invariant feature amount map that includes different types of sensors with various levels of resolution and data types that can be used for a wide range of vehicles. In some embodiments, the feature amount extraction NN 20 is configured to use information such as the time of day, weather conditions, or other information related to the image uploaded using the GUI 24 in order to generate the target invariant feature amount map. The use of additional information in order to generate the target invariant feature amount map helps to remove variables within the image and generate a target invariant feature amount map that most closely matches the target inputted using the GUI 24. In some embodiments, the feature amount extraction NN 20 is configured to exclude confidential information from the target invariant feature amount map.

[0035] The server 18 also includes a template storage section 21 configured to store the target invariant feature quantity map generated by the feature quantity extraction NN 20. In some embodiments, the template storage section 21 is configured to store the target invariant feature quantity map together with the weights of the parameters utilized within the similarity estimation NN 30 or the feature quantity extraction NN 20 used to generate the target invariant feature quantity map. In some embodiments, the template storage section 21 includes a solid state memory or other suitable memory configuration. In some embodiments, the template storage section 21 is integrated with the storage unit 23. In some embodiments, the template storage section 21 is separate from the storage unit 23.

[0036] The server 18 also includes a data collection instruction transmitting section (data collection instruction transmitting section) 22. The data collection instruction transmitting section 22 is configured to transmit a data collection instruction to a vehicle including the ECU 4. In some embodiments, the data collection instruction is transmitted wirelessly. In some embodiments, the data collection instruction is transmitted via a wired connection. In some embodiments, the data collection instruction is transmitted to a single vehicle. In some embodiments, the data collection instruction is transmitted to multiple vehicles. In some embodiments, the data collection instruction is transmitted to vehicles within a geographic location where the target is expected to be found. The data collection instruction includes information about the target based on information received through the GUI 24. In some embodiments, the data collection instruction includes the target invariant feature quantity map. In some embodiments, the data collection instruction includes the weights of the similarity estimation NN 30 or the feature quantity extraction NN 20 used to generate the target invariant feature quantity map. In some embodiments, the data collection instruction transmitting section 22 is configured to transmit the data collection instruction on multiple frequencies. In some embodiments, the data collection instruction transmitting section 22 is configured to transmit the data collection instruction on a single frequency.

[0037] The server 18 also includes a storage unit 23 configured to store information from the GUI 24, as well as input values such as information from the target tracking selection section 19 and weights for the feature quantity extraction NN 20. In some embodiments, the storage unit 23 is configured to store all of the images uploaded using the GUI 24. In some embodiments, the storage unit 23 is configured to store images or cutouts of portions of the images uploaded using the GUI 24. In some embodiments, the storage unit 23 is configured to store the collected data received from the ECU 4. In some embodiments, the storage unit 23 is configured to store the collected data between a given period of time. In some embodiments, the storage unit 23 is configured to store the collected data until the user takes the collected data out. In some embodiments, the storage unit 23 is configured to store the collected data until the additional storage space within the storage unit 23 falls below a storage threshold. In some embodiments, the storage unit 23 includes a solid state memory or other suitable memory configuration.

[0038] The server 18 also includes a collected data storage 25 configured to receive collected data from the ECU 4. The collected data storage 25 is accessible by the user in order to enable the user to retrieve the collected data. In some embodiments, the collected data storage 25 is configured to store the collected data together with information of the time and place associated with the time point at which the collected data was detected. In some embodiments, the collected data storage 25 is configured to store the collected data in association with the corresponding data collection instruction. In some embodiments, the collected data storage 25 is configured to, in response to the reception of collected data based on a data collection instruction associated with the user inputting this information to the GUI 24, initiate a signal for notifying the user of the reception of the collected data. In some embodiments, the notification to the user includes an alert to the user. In some embodiments, the alert is sent to the user's mobile device using a transmitter within the server 18. In some embodiments, a signal is sent to an external transmitter in order to alert the user. In some embodiments, the alert includes a sound alert or a visual alert. In some embodiments, the collected data storage 25 sends the collected data to the storage unit 23 for longer term storage. In some embodiments, the collected data storage 25 includes a solid state memory or another suitable memory configuration. In some embodiments, the collected data storage 25 is integrated with the storage unit 23 and the template storage 21. In some embodiments, the collected data storage 25 is separate from at least one of the storage unit 23 and the template storage 21. In some embodiments, the collected data storage 25 has a database such as a relational database, a NoSQL database, and a key-value store (KVS) database, in order for the user and the tracking target selection unit 19 to be able to retrieve data and select with an inquiry condition.

[0039] By using the vehicle data collection system 100, a user can provide a specific target that should be searched for using only the GUI 24 without having to script search codes that can be used by the ECU 4 and the server 18. By using the invariant feature map as a means for defining a target and for defining collected data, the ECU 4 can perform an accurate similarity analysis with respect to a uniquely requested target. That is, by comparing the invariant feature map based on camera 5 and other sensor 6 data to a target invariant feature map, the risk of a false start and / or false launch is reduced. Also, the feature extraction NN 20 is not dependent on the type of sensors within a different vehicle that has the ability to receive data collection instructions from the server 18, which helps to increase the chances of finding a requested target while minimizing the labor for drafting a program or code associated with the requested target. Also, the feature extraction NN 20 is not dependent on the type of sensors within the same vehicle, which helps to share the same template data for retrieving the same target from data from different sensors within the vehicle system VS, such as a front camera and a rear view camera, so that, for example, a target can be retrieved from data from a rear camera even if the server 18 only has template data generated from a front camera. In addition, the vehicle data collection system 100 can also reduce the transmission of confidential information or personal information for the purpose of reducing or avoiding the infringement of privacy in the process of collecting data associated with a data collection instruction.

[0040] Figure 2 FIG. 20 is a flowchart of a method 200 of generating an invariant feature map according to some embodiments. The method 200 can be used in the vehicle data collection system 100 for generating an invariant feature map and a target invariant feature map. In the method 200, the invariant feature map is generated from, for example, data from the camera 5 or other sensor 6 (200). The data from the camera 5 or other sensor 6 (200) can be received by the GUI 24 (200). The data from the camera 5 or other sensor 6 (200) can be received by the ECU 4 (200). The data from the camera 5 or other sensor 6 (200) can be received by the server 18 (200). Figure 1 Figure 1 Figure 1 ​​) received image data generation. In some embodiments, the feature quantity extraction NN 20 and the feature quantity extraction NN 14 are implemented using the same algorithm and weight data to increase the similarity of the invariant feature quantity map generated when the detected object matches the target. By using the same algorithm and weight data, sometimes the reduction is not initiated and / or is not initiated as compared to other approaches using different algorithms or weight data. In some embodiments, the feature quantity extraction NN 14 and the feature quantity extraction NN 20 are implemented using the same algorithm and different weight data to increase the similarity of the invariant feature quantity map generated when the detected object matches the target. In this case, the weight data within the vehicle is optimized for the HW within the particular vehicle system VS, which further suppresses information that is dependent on or more sensitive to the HW within the VS. In some embodiments, the feature quantity extraction NN 14 and the feature quantity extraction NN 20 are implemented using different algorithms and weight data to increase the similarity of the invariant feature quantity map generated when the detected object matches the target. In this case, the algorithms within the vehicle are optimized for the HW within the particular vehicle system VS, which further suppresses information that is dependent on or more sensitive to the HW within the vehicle system VS.

[0041] The NN is a trained NN that extracts the features of one or more objects or scenes within the image data based on the received attribute data. Also, the NN can be made to produce an invariant feature quantity map for comparison between the detected object and the target. Figure 1 ) received image data generation. In some embodiments, the feature quantity extraction NN 20 and the feature quantity extraction NN 14 are implemented using the same algorithm and weight data to increase the similarity of the invariant feature quantity map generated when the detected object matches the target. By using the same algorithm and weight data, sometimes the reduction is not initiated and / or is not initiated as compared to other approaches using different algorithms or weight data. In some embodiments, the feature quantity extraction NN 14 and the feature quantity extraction NN 20 are implemented using the same algorithm and different weight data to increase the similarity of the invariant feature quantity map generated when the detected object matches the target. In this case, the weight data within the vehicle is optimized for the HW within the particular vehicle system VS, which further suppresses information that is dependent on or more sensitive to the HW within the VS. In some embodiments, the feature quantity extraction NN 14 and the feature quantity extraction NN 20 are implemented using different algorithms and weight data to increase the similarity of the invariant feature quantity map generated when the detected object matches the target. In this case, the algorithms within the vehicle are optimized for the HW within the particular vehicle system VS, which further suppresses information that is dependent on or more sensitive to the HW within the vehicle system VS.

[0042] Figure 3 is a flowchart of a method 300 of determining similarity in some embodiments. The method 300 can be used in the vehicle data collection system 100 ( Figure 1) in the method 300. In the method 300, for example, the invariant feature map from the invariant feature extraction NN 14 ( Figure 1 ) is received by the NN. In some embodiments, the template data corresponds to the target invariant feature map generated by the invariant feature extraction NN 20 ( Figure 1 ). The NN is a NN trained to compare the template data to the invariant feature map and determine how similar the invariant feature map is to the template data. Figure 1

[0043] The NN is a similarity estimation NN, for example, the similarity estimation NN 30 ( Figure 1 ). By using the similarity estimation NN, the similarity estimation has less false positives and negatives than if a rule-based algorithm were used for similarity estimation. By using the similarity estimation NN, the collected data with low similarity is not transmitted to the server, so the amount of data transmitted to the server, for example, the server 18 ( Figure 1 ) is reduced compared to a method that does not include the similarity estimation NN. The reduction in data transmission helps reduce the processing power in the vehicle that includes the similarity estimation NN. Also, in a hybrid vehicle or an electric vehicle, the reduction in data transmission extends the charge in the battery of the vehicle, which in turn provides an extension in the driving distance of the vehicle.

[0044] Figure 4 is a flowchart of a method 400 of using a vehicle data collection system of some embodiments. The method 400 can be implemented by the vehicle data collection system 100 ( Figure 1 ) and the elements used to implement the various steps of the method 400 have reference numerals associated with the vehicle data collection system 100. These reference numerals are provided merely to enable the method 400 to be easily understood. Those skilled in the art will recognize that the method 400 is not limited to implementation only on the vehicle data collection system 100 ( Figure 1 ).

[0045] In act 402, a target is selected that should be tracked. In some embodiments, the target is selected based on input from a user into the GUI 24 ( Figure 1 ). In some embodiments, the target is determined based on uploaded images. In some embodiments, the target is selected from a predetermined list of target options available in the GUI, for example, the GUI 24 ( Figure 1 ). In the case where the user has uploaded image data of the target via the GUI 24, the GUI stores the image data via the storage 25. In some embodiments, the target is automatically determined by the target selection unit 19 without a request from the user via the GUI 24. ​

[0046] In action 404, the data has already been stored in the data collection storage unit, for example, data collection storage unit 25. Figure 1 The collected data is compared with the target. Before action 404, the data has been collected by the data collection method described above or any other data collection method. In some embodiments, image data corresponding to the target is selected based on information provided by the user via GUI 24. For example, when the user provides information such as time, weather, object level, and scene, the data is selected by querying and retrieving data from the database in storage unit 25. In some embodiments, the target selection unit 19 also has another neural network (NN) that classifies whether the image data matches the conditions provided by the user via GUI 24, and the target selection unit 19 selects only the data whose matching conditions are determined by the NN. In some embodiments, the target selection unit 19 has a target data prediction unit unit that predicts the target data that the user may select in the future based on the user's previous request history and information in the database in storage unit 25, and the prediction unit selects the predicted data as the selected data. In some embodiments, action 404 also includes the step of sending which data from the target should be selected as part of the data collection instruction.

[0047] In action 406, an invariant feature mapping is generated based on the selected target data and stored in the template storage unit, for example, template storage unit 21. Figure 1 Target-invariant feature maps, also known as target-invariant feature maps, are generated using a trained neural network to extract features of the target for comparison with collected data. In some implementations, target-invariant feature maps are generated based on additional data associated with the target, such as weather conditions or time. In some implementations, target-invariant feature maps are generated in a manner independent of the type of data or the sensors used to aggregate (collect) the collected data.

[0048] In action 408, the data collection instructions and template data are sent to the vehicle system, such as the vehicle system VS (Vehicle System Visualization). Figure 1) transmission. The data collection instruction instructs the vehicle system to collect data and compare the collected data to template data based on target invariant features. In some embodiments, the data collection instruction also includes priority information. In some embodiments, the data collection instruction also includes weights for the NN algorithm. In some embodiments, the data collection instruction includes information for breaching protection of confidential data collected in the data collection. In some embodiments, the data collection instruction also includes a time period during which data should be collected. In some embodiments, the data collection instruction is transmitted to a single vehicle. In some embodiments, the data collection instruction is transmitted to multiple vehicles. In some embodiments, the data collection instruction is transmitted wirelessly. In some embodiments, the data collection instruction is transmitted via a wired connection.

[0049] In act 410, data is collected by sensors within the vehicle. In some embodiments, data is collected periodically by the sensors at regular intervals. In some embodiments, data is collected by the sensors event-driven, such as when an impact sensor detects that the vehicle has collided with something while driving or parked. In some embodiments, the sensors include cameras, LiDAR, RADAR, electric wave sensors, or other suitable sensors. In some embodiments, the sensors include camera 5 and other sensors 6 Figure 1 ).

[0050] In act 412, the sensor data is stored in a storage, such as data storage 8 Figure 1 ). The sensor data stored in the storage is continuously updated as more sensor data is sent to the storage. In some embodiments, the storage includes an algorithm for determining when the sensor data stored in the storage should be deleted or overwritten.

[0051] In act 414, the sensor data is analyzed for detecting objects acquired by the sensor data. In some embodiments, the sensor data is analyzed using object detection 1 Figure 1 ). The detected objects are identified and classified based on object type, location of the object, proximity of the object relative to the sensor, or other suitable classification criteria. In some embodiments, the objects are detected using a trained NN, such as object detection NN 9, e.g., Single Shot Detector (SSD) or Faster R-CNN Figure 1 ). In some embodiments, object detection 1 detects objects every time data is received from the sensors. In some embodiments, object detection 1 detects objects only if instructed to do so by decision 15. In some embodiments, act 414 of detecting objects is skipped.

[0052] In act 416, an invariant feature amount map is generated based on the extracted features of the detected object. The invariant feature amount map is generated using a NN, such as the feature amount extraction NN 14. Figure 1 In some embodiments, the invariant feature amount map in act 416 is generated by a NN using the same algorithm as the NN used in act 406 to generate the target feature amount map. In some embodiments, the weights from act 406 are provided to the NN used in act 416 based on the data collection instruction. In some embodiments, the feature amount extraction section 15 extracts the invariant feature amount map each time data is received from the sensor. In some embodiments, the feature amount extraction section 15 extracts the invariant feature amount map only if the determination section 15 instructs the feature amount extraction section 15 to extract the invariant feature amount map. In some embodiments, the feature amount extraction section 15 extracts the invariant feature amount map only from the cropped rectangular region within the image data corresponding to the location of the object detected by the object detection section 1, whereby the computational workload and memory bandwidth workload on the ECU 4 are reduced. In some embodiments, the feature amount extraction section 15 extracts the invariant feature amount map from the entire image so as to enable enhancement of the information used to search for a target within the invariant feature amount map.

[0053] In act 418, a data collection instruction is received together with the template data. In some embodiments, the data collection instruction is received by a reception section, such as the reception section 16 Figure 1 In some embodiments in which the data collection instruction includes weights, the weights are provided to the NN performing act 416 prior to the start of act 416, and act 416 is performed by using the received weights for the feature amount extraction NN 14. In some embodiments in which the data collection instruction includes weights, the weights are provided to the similarity estimation NN 30 performing act 420, and act 420 is performed by using the received weights for the similarity estimation NN 30. The data collection instruction and the template data are stored, for example, in the template storage section 17, for comparison with the invariant feature amount map generated in act 416.

[0054] In act 420, a similarity between the template data from the data collection instruction and the invariant feature amount map from act 416 is determined. The similarity is determined using a NN, such as the similarity estimation NN 30 Figure 1

[0055] ​In act 422, a decision is made as to whether the similarity decided in act 420 exceeds a predetermined threshold. In response to a decision that the similarity exceeds the predetermined threshold, sensor data associated with the invariant feature map is acquired from a storage in which the sensor data was stored in act 412. In some embodiments, the sensor data includes data acquired during a period before and after the data used to generate the invariant feature map was acquired. In some embodiments, the sensor data is associated with location and time data. The sensor data and all optional additional information is then transmitted to a server, such as server 18 Figure 5 ) In some embodiments, the sensor data is transmitted wirelessly. In some embodiments, the sensor data is transmitted via a wired connection. In some embodiments, the sensor data is subjected to post-processing, such as data compression, resizing, cropping, etc. before being uploaded. In response to a decision that the similarity is not likely to exceed the predetermined threshold, the sensor data associated with the invariant feature map is discarded or deleted.

[0056] In act 426, the sensor data is received by the server.

[0057] In act 428, the received sensor data is stored, for example, in data store 25 Figure 1 ).

[0058] Those skilled in the art will recognize that the order of the acts of method 400 can be changed in some embodiments. For example, in some embodiments, act 406 is performed before act 404. In some embodiments, at least one act of method 400 is omitted. For example, in some embodiments, act 428 is omitted and the received data is transmitted directly to the user. In some embodiments, additional acts are included. For example, in some embodiments, an alert or notification is transmitted to the user in response to the receipt of sensor data corresponding to a target requested by the user. In some embodiments, multiple acts are combined. For example, in some embodiments, acts 420 and 422 are performed in conjunction.

[0059] Figure 2is a block diagram of a system for implementing a vehicle data collection system according to some embodiments. In system 500, there is included a hardware processor 502, and a non-transitory computer-readable storage medium 504 encoded (i.e., storing) computer program code 506 (i.e., a set of instructions executable by a computer). The computer-readable storage medium 504 is also encoded with commands 507 for interacting with external devices, such as a user's mobile device. The processor 502 is electrically coupled to the computer-readable storage medium 504 via a bus 508. The processor 502 is also electrically coupled to an input / output interface 510 via the bus 508. A network interface 512 is also electrically connected to the processor 502 via the bus 508. The network interface 512 is connected to a network 514, such that the processor 502 and the computer-readable storage medium 504 can be connected to external elements via the network 514. The processor 502 is configured to execute the computer program code 506 encoded in the computer-readable storage medium 504 for the purpose of performing some or all of the actions described in system 100 Figure 3 ), method 200 Figure 4 ), method 300 Figure 1 ), or method 400 Figure 2 ), as described above. The computer program code 506 is configured to be executed by the processor 502 for the purpose of performing some or all of the actions described in system 100

[0060] In some embodiments, the processor 502 is a central processing unit (CPU), a multi-processor, a distributed processing system (DPS), a graphics processing unit (GPU), a tensor processing unit (TPU), an application-specific integrated circuit (ASIC), and / or a suitable processing unit.

[0061] In some embodiments, the computer-readable storage medium 504 is an electronic, magnetic, optical, electromagnetic, infrared, and / or semiconductor system (or apparatus or device). For example, the computer-readable storage medium 504 includes semiconductor or solid-state memory, magnetic tape, a removable computer diskette, a random access memory (RAM), a read-only memory (ROM), a rigid magnetic disc, and / or an optical disc, in some embodiments using optical discs, the computer-readable storage medium 504 includes a compact disc readonly memory (CD-ROM), a CD-R / W, and / or a digital versatile disc (DVD).

[0062] In some embodiments, the storage medium 504 stores computer program code 506 configured to cause the system 500 to perform the method 300 or the method 400. In some embodiments, the storage medium 504 also stores information required to perform the actions of the system 100 Figure 3 ), the method 200 Figure 4 ), the method 300 Figure 1 ), or the method 400 Figure 2 ), as described above. The computer program code 506 is configured to be executed by the processor 502 for the purpose of performing some or all of the actions described in system 100Figure 3 ), the method 200 ( Figure 4 ), the method 300 ( Figure 1 ), or the method 400 ( Figure 2 ) during the actions of the system 100 ( Figure 3 ), the method 200 ( Figure 4 ), the method 300 ( Figure 1 ), or the method 400 ( Figure 2 ). The group of information includes, for example, the sensor data parameter 516, the template data parameter 518, the attribute parameter 900, the invariant feature amount map parameter 520, the similarity parameter 522, and / or the executable command for performing the actions of the system 100 (

[0063] In some embodiments, the storage medium 504 stores a command 507 for interfacing with an external device. The command 507 enables the processor 502 to generate a command that can be read by the external device for efficiently implementing the actions of the system 100 ( Figure 3 ), the method 200 ( Figure 4 ), the method 300 ( Figure 1 ), or the method 400 ( Figure 2 ).

[0064] The system 500 includes an input / output interface (I / O) 510. The input / output interface 510 is coupled with an external circuit. In some embodiments, the input / output interface 510 includes a keyboard, a keypad, a mouse, a trackball, a touchpad, and / or a cursor direction key for communicating information and instructions to the processor 502.

[0065] The system 500 also includes a network interface 512 coupled with the processor 502. The network interface 512 enables the system 500 to communicate with a network 514 connected with one or more other computer systems. The network interface 512 includes, for example, a wireless network interface such as BLUETOOTH (registered trademark), WIFI (registered trademark), WIMAX (registered trademark), GPRS, or WCDMA (registered trademark), or a wired network interface such as ETHERNET (registered trademark), USB, or IEEE-1394. In some embodiments, the actions of the system 100 ( Figure 3 ), the method 200 ( Figure 4 ), the method 300 ( ​ ), or the method 400 ( ​ ) are implemented in two or more systems 500, and information is exchanged between the different systems 500 via the network 514.

[0066] One aspect of the specification relates to a vehicle data collection system. The vehicle data collection system includes an on-board sensor. The vehicle data collection system also includes a non-transitory computer-readable medium configured to store commands. The vehicle data collection system includes a processor coupled to the non-transitory computer-readable medium. The processor is configured to execute the commands to generate an invariant feature map using a first neural network. The processor is also configured to execute the commands to compare the invariant feature map to template data received from a server and determine a similarity between the invariant feature map and the template data. The processor is also configured to execute the commands to determine whether the determined similarity exceeds a predetermined threshold. The processor is also configured to execute the commands to instruct a communication component to transmit sensor data to the server in response to a determination that the determined similarity exceeds the predetermined threshold. In some embodiments, the processor is also configured to execute the commands to detect an object within sensor data acquired by the on-board sensor and generate the invariant feature map based on the detected object. In some embodiments, the on-board sensor includes at least one of a camera, a light detection and ranging (LiDAR) sensor, and a radio detection and ranging (RADAR) sensor. In some embodiments, the template data includes a target invariant feature map generated based on a target. In some embodiments, the processor is also configured to execute the commands to remove confidential information from the sensor data prior to transmitting the sensor data to the server. In some embodiments, the template data includes weights for the neural network, and the processor is configured to execute the commands to generate the invariant feature map using the first neural network with the weights. In some embodiments, the template data includes weights for the neural network, and the processor is also configured to execute the commands to determine the similarity using a second neural network with the weights.

[0067] One aspect of the present disclosure relates to a vehicle data collection system. The vehicle data collection system includes a graphical user interface (GUI) configured to receive target information from a user. The vehicle data collection system also includes a non-transitory computer readable medium configured to store commands. The vehicle data collection system also includes a processor coupled to the non-transitory computer readable medium. The processor is configured to execute the commands to receive the target information from the GUI. The processor is also configured to execute the commands to generate a target-invariant feature map using a first neural network based on the target information. The processor is also configured to execute the commands to generate a data collection instruction based on the target-invariant feature map. The processor is also configured to execute the commands to instruct a transmission unit to transmit the data collection instruction to at least one vehicle. The processor is also configured to execute the commands to receive sensor data from the at least one vehicle in response to the data collection instruction. The processor is also configured to execute the commands to report the sensor data to the user. In some embodiments, the target information includes an image. In some embodiments, the processor is configured to execute the commands to instruct the transmission unit to transmit the data collection instruction to a plurality of vehicles. In some embodiments, the processor is configured to execute the commands to generate the data collection instruction including a priority of a data requirement from the vehicle. In some embodiments, the processor is configured to execute the commands to generate the data collection instruction including a command for preventing the vehicle from removing confidential information from the sensor data based on an identity of the user. In some embodiments, the processor is configured to execute the commands to report the sensor data to the user with at least one of time and location information associated with the sensor data. In some embodiments, the processor is configured to execute the commands to report the sensor data to the user by generating an alert on a mobile device operated by the user.

[0068] At least one aspect of the present specification relates to a method of using a vehicle data collection system. The method includes a step of collecting sensor data using at least one on-board sensor. The method also includes a step of generating an invariant feature map using a first neural network. The method further includes a step of comparing the invariant feature map to template data received from a server and determining a degree of similarity between the invariant feature map and the template data. The method includes a step of determining whether the determined degree of similarity exceeds a predetermined threshold. The method also includes a step of transmitting the sensor data to the server in response to a determination that the determined degree of similarity exceeds the predetermined threshold. In some embodiments, the method further includes a step of detecting an object within the sensor data, where the step of generating the invariant feature map is based on the detected object. In some embodiments, the on-board sensor includes at least one of a camera, a light detection and ranging (LiDAR) sensor, and a radio detection and ranging (RADAR) sensor. In some embodiments, the template data includes a target invariant feature map generated based on a target. In some embodiments, the method further includes a step of removing confidential information from the sensor data prior to transmitting the sensor data to the server. In some embodiments, the template data includes weights for the neural network, and the step of generating the invariant feature map includes a step of using the weights within the first neural network.

[0069] The foregoing summary of features has been presented with respect to the aspects of the present disclosure in a manner that enables a person skilled in the art to better understand the aspects of the present disclosure. Those skilled in the art will recognize, or be able to ascertain using no more than routine experimentation, many equivalents to the specific aspects described herein. It will be understood that these equivalents have not been shown here by way of illustrating the aspects of the present disclosure, and are no less within the spirit and scope of the present disclosure. It is therefore to be understood that all such equivalents are intended to be included herein.

Claims

1. A vehicle data collection system comprising: an in-vehicle sensor; a non-transitory computer-readable medium configured to store a command; and a processor connected to the non-transitory computer-readable medium, the processor configured to execute the command for: generating, using a first neural network, an invariant feature map from sensor data, the invariant feature map being abstracted information related to feature information of an object and / or a scene within an image that can be used for recognizing a specific object and information related to a hardware configuration of the in-vehicle sensor being removed; comparing the invariant feature map with template data received from a server to determine a degree of similarity between the invariant feature map and the template data; determining whether the determined degree of similarity exceeds a predetermined threshold; and in response to a determination that the determined degree of similarity exceeds the predetermined threshold, instructing a transmission unit to transmit the sensor data to the server.

2. The vehicle data collection system according to claim 1, the processor further configured to execute the command for detecting an object within sensor data acquired by the in-vehicle sensor; and generate the invariant feature map based on the detected object.

3. The vehicle data collection system according to claim 1, the in-vehicle sensor includes at least one of a camera, a light detection and ranging sensor (LiDAR sensor), and a radio detection and ranging sensor (RADAR sensor).

4. The vehicle data collection system according to any one of claims 1 to 3, the template data includes a target invariant feature map generated based on a target.

5. The vehicle data collection system according to any one of claims 1 to 3, the processor further configured to execute the command for removing confidential information from the sensor data before transmitting the sensor data to the server.

6. The vehicle data collection system according to any one of claims 1 to 3, the template data includes weights for a neural network, the processor configured to execute the command for generating the invariant feature map using the first neural network with the weights.

7. The vehicle data collection system according to any one of claims 1 to 3, the template data includes weights for a neural network, the processor further configured to execute the command for determining the degree of similarity using a second neural network with the weights.

8. A vehicle data collection system comprising: a graphical user interface (GUI) configured to receive target information from a user; a non-transitory computer-readable medium configured to store a command; and a processor connected to the non-transitory computer-readable medium, the processor configured to execute the command for: receiving target information from the GUI; ​ ​ ​ generating a target-invariant feature map using a first neural network based on the target information, the target-invariant feature map being abstract information in which feature information related to an object and / or a scene within an image that can be used for identifying a specific object is abstracted and information related to a hardware configuration of a sensor within a vehicle is removed; generating a data collection instruction based on the target-invariant feature map; commanding a transmission unit to transmit the data collection instruction to at least one vehicle, the at least one vehicle comparing the target-invariant feature map with template data in response to the data collection instruction, determining a degree of similarity between the target-invariant feature map and the template data, determining whether the determined degree of similarity exceeds a predetermined threshold, and transmitting sensor data in response to a determination that the determined degree of similarity exceeds the predetermined threshold; receiving the sensor data from the at least one vehicle; and reporting the sensor data to the user.

9. The vehicle data collection system according to claim 8, the target information including an image.

10. The vehicle data collection system according to claim 8, the processor configured to execute the command for commanding the transmission unit to transmit the data collection instruction to a plurality of vehicles.

11. The vehicle data collection system according to any one of claims 8 to 10, the processor configured to execute the command for generating a data collection instruction including a priority of a data request from the vehicle.

12. The vehicle data collection system according to any one of claims 8 to 10, the processor configured to execute the command for generating the data collection instruction including a command for preventing the vehicle from removing confidential information from the sensor data based on an identity of the user.

13. The vehicle data collection system according to any one of claims 8 to 10, the processor configured to execute the command for reporting the sensor data to the user together with at least one of time and place information associated with the sensor data.

14. The vehicle data collection system according to any one of claims 8 to 10, the processor configured to execute the command for reporting the sensor data to the user by generating an alert to a mobile device operated by the user.

15. A method of using a vehicle data collection system, comprising: collecting sensor data using at least one vehicle-mounted sensor; generating an invariant feature map from the sensor data using a first neural network, the invariant feature map being abstract information in which feature information related to an object and / or a scene within an image that can be used for identifying a specific object is abstracted and information related to a hardware configuration of the at least one vehicle-mounted sensor is removed; comparing the invariant feature map with template data received from a server, determining a degree of similarity between the invariant feature map and the template data; determining whether the determined degree of similarity exceeds a predetermined threshold; and ​ ​ in response to a determination that the determined similarity exceeds a predetermined threshold, transmitting the sensor data to the server.

16. The method of using a vehicle data collection system of claim 15, generating the invariant feature map further includes generating the invariant feature map based on a detected object within the sensor data.

17. The method of using a vehicle data collection system of claim 15, the on-board sensor includes at least one of a camera, a light detection and ranging (LiDAR) sensor, and a radio detection and ranging (RADAR) sensor.

18. The method of using a vehicle data collection system of any one of claims 15-17, the template data includes a target invariant feature map generated based on a target.

19. The method of using a vehicle data collection system of any one of claims 15-17, further including removing confidential information from the sensor data prior to transmitting the sensor data to the server.

20. The method of using a vehicle data collection system of any one of claims 15-17, the template data includes weights for a neural network, and generating the invariant feature map includes using the weights within the first neural network.

Images