Systems and methods for assisting object detection

Abstract

The support system receives sensor data, including data captured by a sensor unit mounted on the vehicle, and obtains first captured data, which is data captured in a first time step, and second captured data, which is data captured in a second time step, from the sensor data. The second time step is the time step following the first time step among a plurality of consecutive time steps. The support system calculates a first feature quantity related to the first captured data and a second feature quantity related to the second captured data, determines whether the relationship between the first feature quantity and the second feature quantity satisfies predetermined conditions, and if the determination result is true, the support system fuses the first captured data with the second captured data and outputs the fused data, which includes the data obtained by fusing the first captured data with the second captured data, to a function that detects objects based on the data input to that function.

Description

Technical Field

[0001] The present invention generally relates to techniques for assisting object detection, for example, for assisting the autonomous driving of trains or other vehicles.

Background Art

[0002] Generally, since the FOV (field of view) of a sensor (typically a camera) is narrow, the sensor cannot capture the entire object. Since the object is partially captured by the sensor, it is difficult for the sensing data (data from the sensor) to be input into a function (such as an algorithm) for detecting the object and for that function to detect the object based on the sensing data.

[0003] Regarding the solution to the problem when the FOV of the sensor is narrow, there are prior art documents such as Patent Document 1 and Patent Document 2.

[0004] Patent Document 1 discloses a vehicle vision system equipped with a plurality of cameras in order to provide a surround view display image with a wider FOV by the plurality of cameras.

[0005] Patent Document 2 discloses a method for presenting a panoramic surround view inside a vehicle.

Prior Art Documents

Patent Documents

[0006]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0007] Patent documents 1 and 2 require additional identical sensors (e.g., additional cameras) to reduce blind spots (in other words, to ensure a wide field of view). This increases the number of components. [Means for solving the problem]

[0008] The object detection support system (hereinafter referred to as the "support system") receives sensor data, including data captured by a sensor unit mounted on the vehicle, and obtains first captured data, which is data captured in the first time step, and second captured data, which is data captured in the second time step, from the sensor data. The second time step is the time step following the first time step among a plurality of consecutive time steps. The support system calculates a first feature quantity, which is a feature quantity relating to the first captured data, and a second feature quantity, which is a feature quantity relating to the second captured data. The support system determines whether the relationship between the first feature quantity and the second feature quantity satisfies predetermined conditions. If the result of the determination is true, the support system performs a fusion process, which includes fusing the first captured data with the second captured data, and outputs the fused data, which includes the data obtained by fusing the first captured data with the second captured data, to the object detection system, which is a function that detects objects based on the data input to the function. [Effects of the Invention]

[0009] According to the present invention, it is possible to improve the accuracy of object detection without the need for additional identical sensors. [Brief explanation of the drawing]

[0010] [Figure 1] An example of the overall system configuration, including the sensor system according to the first embodiment, is shown. [Figure 2] An example of the processing flow according to the first embodiment is shown. [Figure 3] A detailed example of S205 in Figure 2 is shown. [Figure 4] An example of the overall system configuration, including the sensor system according to the second embodiment, is shown. [Figure 5] An example of the processing flow according to the second embodiment is shown. [Figure 6] A detailed example of S505 in Figure 5 is shown. [Figure 7] A schematic example of the fusion process is shown. [Figure 8] A schematic example of the fusion process is shown. [Figure 9] A schematic example of the fusion process is shown. [Figure 10] This example shows the relationship between a train operation control system and object detection. [Modes for carrying out the invention]

[0011] In the following description, "interface device" can refer to one or more interface devices. These one or more interface devices may be at least one of the following: *One or more I / O (input / output) interface devices. An I / O interface device is an interface device to at least one of an I / O device or a remote display computer. The I / O interface device to the display computer may also be a communication interface device. At least one I / O device can be either an input device such as a keyboard or a pointing device, or an output device such as a display device, which are both user interface devices. *One or more communication interface devices. One or more communication interface devices may be one or more communication interface devices of the same type (e.g., one or more NICs (Network Interface Cards)) or two or more communication interface devices of different types (e.g., a NIC and an HBA (Host Bus Adapter)).

[0012] Furthermore, in the following description, "memory" is one or more memory devices as examples of one or more storage devices, and typically is a main storage device. At least one memory device in the memory may be a volatile memory device or a non-volatile memory device.

[0013] Furthermore, in the following description, "persistent storage device" may be one or more persistent storage devices as examples of one or more storage devices. A persistent storage device may typically be a non-volatile storage device (e.g., an auxiliary storage device), specifically, for example, an HDD (Hard Disk Drive), an SSD (Solid State Drive), an NVME (Non-Volatile Memory Express) drive, or an SCM (Storage Class Memory).

[0014] Furthermore, in the following description, "storage device" may be at least memory, or the memory of a persistent storage device.

[0015] Furthermore, in the following description, “processor” may refer to one or more processor devices. At least one processor device may typically be a microprocessor device such as a CPU (Central Processing Unit), or another type of processor device such as a GPU (Graphics Processing Unit). At least one processor device may be a single-core processor device or a multi-core processor device. At least one processor device may be a processor core. At least one processor device may be a broad processor device such as a circuit as a collection of gate arrays based on a hardware description language that performs some or all of the processing (e.g., FPGA (Field-Programmable Gate Array), CPLD (Complex Programmable Logic Device), or ASIC (Application Specific Integrated Circuit)).

[0016] Furthermore, in the following explanation, functions may be described using expressions such as "yyy unit," but these functions may be implemented by one or more computer programs executed using a processor, or by one or more hardware circuits (e.g., FPGA or ASIC), or based on a combination thereof. When a function is implemented by a program executed using a processor, the predetermined processing is performed using memory and / or interface devices as appropriate, so the function may be at least a part of the processor. The processing described with a function as the subject may be processing performed by the processor or a device containing such a processor.

[0017] Furthermore, the program may be installed from a program source. The program source may be, for example, a program distribution computer or a computer-readable recording medium (e.g., a non-transitory recording medium). The description of each function is an example, and a plurality of functions may be aggregated into one function, or one function may be divided into a plurality of functions.

[0018] Furthermore, in the following description, when describing elements of the same type without distinction, common reference numerals among the reference numerals may be used, and when elements of the same type are distinguished, reference numerals may be used.

[0019] Hereinafter, some embodiments of the present invention will be described with reference to the drawings.

[0020] [First Embodiment] FIG. 1 shows an example of the configuration of an entire system including a sensor system according to the first embodiment.

[0021] The sensor system 100 is a physical computer system (e.g., a motherboard) in the present embodiment, but may be a logical computer system (e.g., as an example of a physical computer system, a system as a cloud computing service provided by a cloud infrastructure). The sensor system 100 includes an interface device 110, a storage device 120, and a processor 130.

[0022] Data is input from the camera unit 190 and the accelerometer unit 180 via the interface device 110. The camera unit 190 is the camera itself or a device having a camera. The accelerometer unit 180 is the accelerometer itself or a device having an accelerometer.

[0023] The data is output via the interface device 110. The output data is input to the object detection system 141. The object detection system 141 may be an external computer system to the sensor system 100, or it may be a function within the sensor system 100. The functions in the sensor system 100 (or other computer system) may be implemented by one or more computer programs executed by a processor, by one or more hardware circuits (e.g., FPGA or ASIC), or a combination thereof.

[0024] The storage device 120 stores computer programs executed by the processor 130 and data that is input and output between the sensor system 100 and the computer.

[0025] The processor 130 is coupled to the interface device 110 and the storage device 120. The processor 130 loads and executes computer programs, thereby realizing one or more functions.

[0026] The processor 130 obtains images (e.g., RGB color images) 101A and 101B from the data input from the camera unit 190 at two consecutive time steps (hereinafter referred to as t and t+1). Image 101A at time step t may be referred to as "image t" or "frame t". Image 101B at time step t+1 may be referred to as "image t+1" or "frame t+1".

[0027] The processor 130 also identifies the displacement 102 (velocity or displacement) between timesteps t and t+1 from the data input from the accelerometer unit 180 (data representing the measured acceleration).

[0028] Processor 130 processes image t, image t+1, and movement displacementBased on 102, a fusion process is performed, including the generation of fused data 116. The fused data 116 is the result of fusing the input image t and image t+1. The processor 130 transfers the generated fused data 116 to the object detection system 141. The object detection system 141 is an image-based object detection system that detects objects based on the fused data 116 (for example, it may use a machine learning model such as a neural network to detect objects).

[0029] Figure 2 shows an example of the processing flow according to the first embodiment. In this description, "object" may mean an object in an image. "Object pixel" may be pixel information about a detected object in the image (e.g., information including color information such as RGB information and information representing the object's coordinates). "Object mask" may include information representing the type of object detected in the image (e.g., information that the object pixel is green, or information about the object's label) and information representing the coordinates of the object on the image.

[0030] The camera unit 190 is a vehicle to Preparation Erare The shooting direction of the camera unit 190 may be the same as the direction of travel of the vehicle. The vehicle may be a railway vehicle (for example, a railway vehicle running at the front of a train) or another type of vehicle such as an automobile. The sensor system 100 is the vehicle to Preparation Erare In some cases, it may be a remote system for the vehicle. In this embodiment, both the camera unit 190 and the sensor system 100 are located on the railway vehicle. to Preparation Erare It is assumed that this is the case.

[0031] Processor 130 performs processing to enhance image t (image captured at time step t) (S201). Processor 130 extracts SIFT (Scale-Invariant Feature Transform) features and optical flow from the enhanced image t (S202).

[0032] The processor 130 also inputs the enhanced image t into semantic segmentation (e.g., semantic segmentation for an unmanned train) (S203) to obtain a set of pixels (typically a set of color pixels) with segmented object masks (a mask for each target object).

[0033] Processor 130 also extracts SIFT features and optical flow from image t+1 (S204).

[0034] Furthermore, there is no need to perform any enhancement processing (i.e., S201) on image t+1. This is because, for both image t and t+1, the shooting direction is the same as the direction of the vehicle's movement, and it is expected that all objects that were blurred in image t can be captured in image t+1.

[0035] S204 may be the same process as S202. In S202 and S204, functions for extracting SIFT features and functions for extracting optical flow may be used, respectively. In S202 and S204, features may be extracted by other methods. For example, by inputting image t and image t+1 into a CNN (Convolutional Neural Network) that captures image features, features can be captured for each of image t and image t+1.

[0036] The processor 130 identifies repeating objects by inputting the segmented object mask (a mask for each target object), the SIFT features and optical flow of the enhanced image t (which may include further color features of the enhanced image t), and the SIFT features and optical flow of image t+1 (which may include further color features of image t+1) into the Siamese network (S205). In S205, the processor 130 outputs object mask data representing the non-repeating object mask (object mask for non-repeating objects) and the repeating object mask (object mask for repeating objects). The processor 130 blacks out the repeating objects in image t to prevent the object detection system 141 from repeatedly detecting the same object from the fused data 116 (S206). As a result, the processor outputs image t with the repeating objects blacked out and data representing the repeating object mask. Even if other methods are applied, such as image inpainting using a diffusion / GAN model, instead of repeatedly blacking out objects, the object detection system 141 is expected to avoid repeatedly detecting the same object from the fused data 116.

[0037] Meanwhile, the processor 130 uses the movement displacement 102 between t and t+1 to determine whether the SIFT features and optical flow of the enhanced image t match those of the image t+1 (S207). Specifically, for example, the processor 130 uses the velocity / displacement between t and t+1 and a general optimization algorithm such as a nearest neighbor algorithm using the Euclidean distance between keypoint descriptors to determine whether there are matching features.

[0038] If the result of S207 is false (S207: No), for example, if there are no matching features, or if the number of matching features is less than a predetermined threshold, the processor 130 discards the data generated in S206 (data representing image t having black repeating objects and repeating object masks). As a result, image t+1 becomes the new image t, and image t+1 is used as the new image t, and image t+2 is used as the new image t+1, and the process shown in Figure 2 is executed again. The remaining image t+1 that is not discarded can be input to the object detection system 141. The object detection system 141 can detect objects in the input image t+1.

[0039] If the result of the S207 judgment is true (S207:Yes), for example, if there are matching features, or if the number of matching features is greater than a predetermined threshold, the processor 130 performs stitching of image t and image t+1 based on the matching features (S208). As a result, a single stitched image is output.

[0040] In S208, for example, the processor 130 may continue to receive image t+1, data generated in S206, feature data (acquired in S202 and S204 respectively), and translational displacement 102 (velocity / displacement between t and t+1), and use this received data for image stitching. Specifically, for example, at least the first of the first and second functions may be used. The first function may be a function for stitching image t+1 in the generated data to image t based on SIFT features and optical flow, using a common optimization algorithm such as a nearest neighbor algorithm using Euclidean distance between keypoint descriptors. The second function may be a function for adjusting the stitching result based on the translational displacement 102 to minimize feature errors using homography.

[0041] The processor 130 performs stitching on partial repeating objects in image t+1 according to a general image stitching method (S209), based on the stitched image and the data generated in S205 (e.g., a repeating object mask and features related to the repeating object mask (which may include features related to each object)). As a result, an image containing the stitched objects is produced as the final output of the entire fusion process. The processor 130 inputs the image generated in S209 to the object detection system 141.

[0042] Figure 3 shows a detailed example of S205 in Figure 2.

[0043] The object detection system 141 performs image-based object detection as described above.

[0044] Specifically, for example, the processor 130 extracts corresponding image features based on the pixel positions of the target object mask (data included in the data acquired by S203) and feature data related to image t (data acquired by S202) (S301). As a result, the extracted features are associated with the data representing the target object pixels (pixels of the target object mask). A .

[0045] Next, the processor 130 determines whether there are any image regions with a high degree of feature agreement with respect to image t+1, based on the data generated in S301 and the feature data (data acquired in S204), and outputs data relating to such image regions (regions with features) (S302).

[0046] The processor 130 inputs the data generated by S302 and the data generated by S301 into the same trained convolutional neural layer (e.g., all or part of a Siamese network) to determine whether two object pixels with features belong to the same object (S303 and S304). For example, the object corresponding to the object mask of the data generated by S301 is the fully captured person (the person captured from image t) as illustrated in Figure 3, while the object corresponding to the object mask of the data generated by S302 is partially captured. The data generated by S301 (e.g., data containing the features and coordinates of each object pixel) is input into the convolutional neural layer, and data representing similarity (e.g., a binary probability score of similarity) is output from the convolutional neural layer. Similarly, when the data generated by S302 (e.g., data containing the features and coordinates of each object pixel) is input into the same convolutional neural layer, data representing similarity is output from that convolutional neural layer. The coordinates of an object in image t+1 can be converted by the processor 130 to coordinates in image t based on the identified movement displacement 102 (velocity / displacement between t and t+1). Based on the output in S303 (e.g., binary probability score) and the output in S304 (e.g., binary probability score), the processor 130 can determine whether the object (object mask) is repeatable or non-repeatable, generate data based on the determination result, and output the generated data.

[0047] The above is a description of the first embodiment.

[0048] [Second Embodiment] A second embodiment will be described. In this case, the differences from the first embodiment will be mainly described, and the explanation of points common to the first embodiment will be omitted or simplified.

[0049] Figure 4 shows an example of the overall system configuration including the sensor system according to the second embodiment.

[0050] The hardware configuration of the sensor system 400 may be the same as that of the sensor system 100. However, in this embodiment, the processor 130 implements other functions in addition to or instead of at least some of the functions implemented in the first embodiment.

[0051] Data is input from the LiDAR (Light Detection and Ranging) unit 450 via the interface device of the sensor system 400. The LiDAR unit 450 is either the LiDAR itself or a device equipped with LiDAR. LiDAR is an example of a remote sensor and is sometimes referred to as a TOF (Time-Of-Flight) sensor. In this embodiment, point cloud-based image fusion is performed.

[0052] The processor 130 obtains camera parameters 403 (data containing one or more camera-specific parameters) from the data input from the camera unit 190, similar to images 101A and 101B.

[0053] The processor 130 captures point clouds 401A and 401B corresponding to two consecutive time steps t and t+1 from the data input from the LiDAR unit 450. Hereinafter, point cloud 401A at time step t may be referred to as "point cloud t," and point cloud 401B at time step t+1 may be referred to as "point cloud t+1."

[0054] The processor 130 acquires relative position data (data representing the relative position between the LiDAR unit 450 and the camera unit 190) from the data input from the LiDAR unit 450.

[0055] The processor 130 performs a fusion process, including the generation of fused data 416, based on image t, point cloud t, image t+1, point cloud t+1, camera parameters 403, relative position data 413, and movement displacement 102. The processor 130 transfers the generated fused data 416 to the object detection system 441. The object detection system 441 is a point cloud-based object detection system that detects objects based on the fused data 416 (for example, it may detect objects using a machine learning model such as a neural network). Similar to the object detection system 141, the object detection system 441 may be located inside or outside the sensor system 400.

[0056] Figure 5 shows an example of the processing flow according to the second embodiment.

[0057] The processor 130 inputs the image t and point cloud t into semantic segmentation (e.g., semantic segmentation for an unmanned train) (S503) to obtain a set of pixels with segmented object masks (masks for each target object). The objects are objects on the point cloud t.

[0058] Processor 130 extracts the features of point cloud t (S502) and extracts the features of point cloud t+1. group t and point group Using the displacement 102 between t+1, point group Features and points of t groupThe system determines whether the features of t+1 match (S504). Specifically, for example, the processor 130 uses the velocity / displacement between t and t+1 and a general optimization algorithm such as a nearest neighbor algorithm using the Euclidean distance between keypoint descriptors to determine whether there are matching features. If the result of S504 is false (S504: No), for example, if there are some matching features, or if the number of matching features is less than a predetermined threshold, the processor 130 discards the data (point cloud t with a repeating object mask) and image t generated in S506, which will be described later. As a result, image t+1 becomes a new image t, point cloud t+1 becomes a new point cloud t, and image t+1 is used as a new image t, point cloud t+1 as a new point cloud t, image t+2 as a new image t+1, and point cloud t+2 as a new point cloud t+1, and the process shown in Figure 5 is executed again. The remaining point cloud t+1 that has not been discarded can be input to the object detection system 441.

[0059] The processor 130 projects a mask onto the point cloud based on image t, point cloud t, image t+1, point cloud t+1, the data generated by S501 (data representing the semantically segmented object mask), and image projection parameters (camera parameters 403 and relative position data 413), and identifies repeated objects using a Siamese network (S505). In S505, the processor 130 outputs object mask data representing non-repeating object masks and repeating object masks. The data input to the Siamese network may include reflection values ​​from LiDAR in addition to, or instead of, the color information of each pixel in image t and image t+1. The processor 130 removes point clouds of repeated objects from point cloud t to prevent the object detection system 441 from repeatedly detecting the same object from the fused data 416 (S506). The removal of point clouds of repeated objects is sometimes referred to as masking of point clouds of repeated objects. In other words, S506 can generate a point cloud t with repeating object masks.

[0060] If the result of S504 is true (S504:Yes), for example, if there are matching features, or if the number of matching features is greater than a predetermined threshold, the processor 130 will determine the points based on the matching features. group t and point group Align t+1 (S507).

[0061] In S507, for example, the processor 130 may continue to receive the point cloud t+1, the generated point cloud (data generated in S506), point cloud features (features extracted in S502 and S503), and the translation displacement 102. Specifically, for example, at least one of the first and second functions may be used. The first function may be a function for aligning the point cloud t+1 with the generated point cloud based on the point cloud features using a neural network. The second function may be a function for adjusting the alignment result (aligned point cloud) based on the translation displacement 102 so that the feature error difference is minimized using homography.

[0062] After S507, based on the aligned point cloud and the data generated in S506 (including the repeating object mask), the processor 130 replaces the partially captured repeating objects at time step t+1 with the information objects captured at time step t (information objects at the location (e.g., vehicle position) at time step t+1), and generates a point cloud with stitched objects as the final output of the entire fusion process. The processor 130 inputs the point cloud generated in S508 to the object detection system 441.

[0063] Figure 6 shows a detailed example of S505 in Figure 5.

[0064] The object detection system 441 performs point cloud-based object detection as described above.

[0065] Specifically, for example, the processor 130 projects a mask onto the point cloud (S601) based on the target object mask (data included in the data acquired by S505), image projection parameters (camera parameters 403 and relative position data 413), the point cloud t, and the image t. The purpose of S601 is to project the object mask (data including object type and RGB color information) onto the corresponding points based on the camera parameters 403 and relative position data 413. The camera parameters 403 may include data representing the focal length and central optical system, and the relative position data 413 may include data representing the rotation angle and sensor displacement. S601 generates a point cloud having color and object mask information for each point.

[0066] The processor 130 projects a mask onto the point cloud (S602) based on the image projection parameters (camera parameters 403 and relative position data 413), the point cloud t+1, and the image t+1. S601 and S602 may be performed by the same function. S602 generates a point cloud having color information for each point.

[0067] The processor 130 extracts the target object point cloud from the point cloud generated in S601 using color information (e.g., color features) (S603). Based on the target object point cloud extracted in S603 and the point cloud with color information generated in S602, the processor 130 searches for regions with similar color and point cloud features (matching regions) and generates a matching point cloud with color features that is a point cloud in that region (S604).

[0068] The processor 130 inputs the point clouds generated by S603 and S604 into the same trained convolutional neural layer (e.g., all or part of a Siamese network) to determine whether the two object point clouds with color features belong to the same object (S605 and S606). The point cloud generated by S603 is input into the convolutional neural layer, and the convolutional neural layer outputs data representing similarity (e.g., a binary probability score for similarity). Similarly, the point cloud generated by S604 is input into the same convolutional neural layer, and the convolutional neural layer outputs data representing similarity. Based on the output in S605 (e.g., a binary probability score) and the output in S606 (e.g., a binary probability score), the processor 130 may determine whether the object (object mask) is repeating or non-repeat, generate data based on the determination result, and output the generated data.

[0069] The difference between the convolutional neural layer shown in Figure 6 and the convolutional neural layer shown in Figure 3 lies in the input dimension. The input dimension of the convolutional neural layer shown in Figure 3 may include SIFT features, optical flow, and RGB color features, while the input dimension of the convolutional neural layer shown in Figure 6 may include point positions (e.g., point coordinates) instead of SIFT features and optical flow. The convolutional neural layer can be a sparse convolutional neural layer to reduce computation time.

[0070] The above describes the second embodiment.

[0071] The following is a schematic description with reference to Figures 7-9 of some examples of fusion processing according to the first and second embodiments. These examples may be examples of the application of fusion to an unmanned train. In Figures 7-9, the mesh pattern in image t and image t+1 before fusion represents the sky (background). In Figures 7-9, the object is a person. In Figures 7-9, each point on the person represents a point captured by LiDAR.

[0072] Generally, the purpose of semantic segmentation is to recognize the object type for each pixel on an image based on color input. In an unmanned train, there are common object types that affect safe operation. In S203, processor 130 predicts the type of target object to increase the probability that the target object will be detected after fusion. Examples of target objects may be pedestrians, logs, vehicles (e.g., bicycles, cars, trucks, buses), and animals (e.g., cats, dogs, deer, rabbits).

[0073] The latest position is important as an object position when applied to unmanned trains.

[0074] The first and second cases are shown in the example in Figure 7. These two cases share the commonality that an object in the same corresponding region of image t+1 is partially captured in image t+1 (for the same object, point cloud t corresponds to the entire object, and point cloud t+1 corresponds to a part of the object).

[0075] In image fusion, the entire region corresponding to image t+1 of the enhanced image t is replaced with the measurements of image t+1. In point cloud fusion, the point cloud t+1 corresponding to a partial object is combined with the point cloud corresponding to the rest of the object.

[0076] For example, in the first case, in S205 or S505, it is identified that the same object partially captured in image t+1 (the same object recognized by the Siamese network and the object in image t) is in the same location (in other words, the object has not moved). Therefore, the processor 130 performs object stitching (S209 or S508) without performing an update (S206 or S506).

[0077] On the other hand, in the second case, in S205 or S505, it is identified that the same object partially captured in image t+1 (the same object recognized by the Siamese network and the object in image t) is not in the same position (in other words, the object has moved), and that the object remains in image t after image t+1 has been applied to the entire region of the corresponding region of image t+1 in image t. Therefore, the processor 130 performs an update (S206 or S506) for the remaining object in image t (the portion of the object not in the region corresponding to image t+1) to avoid detecting object repetitions, and performs object stitching (S209 or S508). Specifically, for example, the processor 130 blacks out the portion of the image of the object (or point cloud of the object) in image t that is not in the region corresponding to image t+1 (or deletes the portion of the point cloud), and stitches the portion of the object in image t to the portion of the object in the region corresponding to image t+1.

[0078] As shown in the example in Figure 8, an object in image t is partially captured in image t+1. In S205 or S505, it is identified that the same object partially captured in image t+1 (the same object recognized by the Siamese network and the object in image t) is not in the same position, but that applying image t+1 to the entire area corresponding to image t+1 in image t does not leave any object in image t. Therefore, the processor 130 performs stitching (S209 or S508) without performing an update (S206 or S506).

[0079] As shown in the example in Figure 9, in S205 or S505, it is identified that the same object (recognized by the Siamese network and the same object in image t) that was partially captured in image t+1 is not in the same position, and that the object remains in image t after image t+1 is applied to the region of image t corresponding to image t+1. Therefore, the processor 130 performs an update (S206 or S506) and stitches (S209 or S508) to avoid detecting repeated objects.

[0080] The processing in S208 may be replaced with the following processing instead of the processing described above. For example, the processor 130 may perform stitching based on the translation displacement 102 and then fine-tune the stitching based on the image features. Alternatively, the processor 130 may perform stitching based on the image features and then fine-tune the stitching based on the translation displacement 102. The processing in S208 does not necessarily have to be divided into a first stage and a second stage. The processor 130 may perform the processing in S208 in such a way that the stitching error is as small as possible.

[0081] The processing in S507 may be replaced with the following processing instead of the processing described above. For example, the processor 130 may perform a transformation based on the translation displacement 102 and then fine-tune the alignment based on the point cloud features. Alternatively, the processor 130 may perform alignment based on image features and then fine-tune the alignment based on the translation displacement 102. The processing in S507 does not have to be divided into a first stage and a second stage. The processor 130 may perform the processing in S507 in such a way that the alignment error is minimized.

[0082] Although several embodiments have been described above, these are merely examples to illustrate the present invention and are not intended to limit the scope of the invention to these embodiments alone. The present invention can also be implemented in various other forms.

[0083] The above explanation can be summarized, for example, as follows. The following summary may include supplementary or modified descriptions of the above statement.

[0084] The object detection support system (hereinafter referred to as the "support system") receives sensor data, including data captured by a sensor unit mounted on the vehicle. For example, the interface device 110 receives the sensor data, and the sensor data is stored in the storage device 120. The support system (for example, the processor 130) obtains from the sensor data first captured data, which is data captured in a first time step, and second captured data, which is data captured in a second time step. The second time step is the time step (time step t+1) following the first time step (for example, time step t) among a plurality of consecutive time steps. The support system (for example, sensor system 100 or 400) calculates a first feature quantity, which is a feature quantity relating to the first captured data, and a second feature quantity, which is a feature quantity relating to the second captured data. It then determines whether the relationship between the first feature quantity and the second feature quantity satisfies predetermined conditions. If the result of the determination is true, the support system performs a fusion process, which includes fusing the first captured data with the second captured data, and outputs the fused data, which includes the data obtained by fusing the first captured data with the second captured data, to the object detection system, which is a function that detects objects based on the data input to the function.

[0085] The second acquisition data corresponding to a second time step (e.g., the present) is expected to be covered by the first acquisition data corresponding to a past time step (the first time step), and this fused data includes the data where the second acquisition data has been fused with the first acquisition data. This allows for an expansion of the FOV without additional identical sensor units, thereby improving the accuracy of object detection.

[0086] The determination described above (for example, S207 in Figure 2 or S504 in Figure 5) is based on the vehicle's displacement between the first and second time steps (for example, the displacement identified using a sensor unit such as the accelerometer unit 180). Furthermore, this determination is based on whether the second feature is similar to the first feature. This avoids the fusion process being performed if the second feature is not similar to the first feature.

[0087] The sensor unit may be a camera unit 190. The shooting direction of the camera unit 190 may be the same as the direction of travel of the vehicle. The first capture data may be a first image (e.g., image t), and the second capture data may be a second image (e.g., image t+1). Both the first and second images may be RGB images, other types of color images, or monochrome images. The fused data includes a fused image, which is an image obtained by fusing the second image with a region of the first image corresponding to the second image (e.g., the region corresponding to image t+1). The first feature may include at least one of the SIFT feature and optical flow of the first image. The second feature may include at least one of the SIFT feature and optical flow of the second image. The fusion process may include stitching a portion of an object in the first image with the same object partially captured in the second image. The fused data may be, for example, the fused data 116 described above. In the fused image, the rest of the object is stitched with the object partially captured based on the object in the first image. This can improve the accuracy of object detection.

[0088] The support system may generate first processed data by applying processing using semantic segmentation to the first image or an enhanced first image. Based on the first processed data, a first feature, and a second feature, the support system may perform a tracking process, which is a process for object tracking. If the tracking process identifies repeating objects, the support system may apply an inpainting process to the repeating objects, so that the objects with the inpainting process applied are present in the fused image. This avoids repeatedly detecting the same object. The repeating objects may be objects in the first image, or objects or parts of objects in a region of the first image other than the region corresponding to the second image.

[0089] The support system may apply enhancement processing to at least the first of the two images. This is expected to increase the likelihood of recognizing objects that are clear in the second image but unclear in the first image using semantic segmentation. The enhancement may be GAN-based or based on other methods.

[0090] The sensor unit may be a LiDAR unit 450 configured to capture objects in the direction of the vehicle's movement. The first capture data may be a first point cloud (e.g., point cloud t+1). The second capture data may be a second point cloud (e.g., point cloud t). The first feature may include the feature of the first point cloud. The second feature may include the second point cloud. A "point cloud" may be one or more points. The fused data may include a point cloud obtained by reflecting the first point cloud onto the second point cloud. The fused data can be, for example, the fused data 416 described above. In this way, in the fused data, the point cloud corresponding to a partially captured object is combined with the point cloud corresponding to the remaining part of the object, and as a result, it is expected that the point cloud corresponding to the entire object will be included. Therefore, it is expected that the object detection accuracy will be improved.

[0091] The support system (for example, interface device 110) receives data not only from the LiDAR unit 450 but also from the camera unit 190 installed in the vehicle. The data received from the camera unit 190 is used to create the first image described above. and The second image is included. The first point cloud may be a point cloud that is partially captured in the second image and corresponds to the same object or part of an object as the object captured in the first image. This is expected to improve the accuracy of object detection.

[0092] The support system processes the first image (or an enhanced first image) using semantic segmentation. ) By applying this, first processing data can be generated. Based on the first processing data, image projection data, second point cloud, and second image, the support system can perform a tracking process, which is a process for object tracking. The image projection data may include parameters related to the camera unit 190 (e.g., camera parameters 403) and data representing the relative position between the camera unit 190 and the LiDAR unit 450 (e.g., relative position data 413). The camera parameters 403 may be data entered by the user rather than obtained from data received from the camera unit 190. The relative position data 413 may be data entered by the user rather than obtained from data received from the LiDAR unit 450. Furthermore, if a repeating object is identified by the tracking process, the support system may delete the point cloud corresponding to the repeating object, and the fused data may not include the deleted point cloud. This avoids repeatedly detecting the same object. The repeating object may be an object in the first image, or an object or part of an object in a region of the first image other than the region corresponding to the second image.

[0093] For example, in a second embodiment, the fused data may include a fused image, which is an image obtained by fusing the second image into a region of the first image corresponding to the second image. The fusion process may include stitching a portion of an object in the first image to the same object partially captured in the second image. This is expected to improve the accuracy of object detection.

[0094] As shown in Figure 10, the system may include an object detection system 1020 (e.g., object detection systems 141 and / or 441, or a system including those systems 141 and 441) configured to detect objects from input data, an object detection support system 1010 (e.g., sensor systems 100 and / or 400, or a system including those systems 100 and 400) configured to assist object detection by the object detection system 1020, and a running control system 1000 configured to control the autonomous running of the train 1050 based on the results of object detection by the object detection system 1020. Systems 1000, 1010, and 1020 are configured to control the train 1050 equipped with a sensor unit 1030. to Preparation Rare The sensor unit 1030 may also be a camera unit 190 and / or a LiDAR unit 450. The train 1050 may be an N-car train (where N is a natural number), i.e., N railway cars. The train 1050 may also be an unmanned train.

[0095] Support system (object detection support system) 1010 is train 1050 to Preparation ErareFrom the sensor data, including data captured by the sensor unit 1030, first captured data and second captured data may be obtained, and first and second feature quantities may be calculated. The support system 1010 can determine whether the relationship between the first and second feature quantities satisfies predetermined conditions. If the determination result is true, the support system 1010 may perform a fusion process, including fusing the second captured data with the first captured data, and output fused data, including the data obtained by fusing the second captured data with the first captured data, to the object detection system 1020. The object detection system may perform object detection based on the fused data and output the object detection result (data representing the result) to the driving control system 1000. Based on the result, the driving control system 1000 can control the autonomous driving of the train 1050. Since the accuracy of object detection is expected to improve, the accuracy of autonomous driving control is also expected to improve.

[0096] Point cloud-based fusion and image-based fusion can be used in combination. In this case, the fused data is input to a detection system (e.g., system 1020) which includes an object detection system 441 that is a point cloud-based system and an object detection system 141 that is an image-based system. The detection system can then determine the result of object detection based on the results of systems 141 and 441 and output the determined result.

[0097] In image-based object detection, information representing the type of object may be provided. In point cloud-based object detection, information representing the 3D size and 3D location of the object may be provided.

[0098] According to the embodiments described above, the appearance of a partially captured object in a second image corresponding to a second time step (or a point cloud corresponding to a partially captured object) may be interpolated based on the appearance of the object in a first image corresponding to a first time step (one step prior to the second time step), or a point cloud corresponding to the object.

[0099] The consecutive time steps are not limited to two time steps, but may be three or more. If the consecutive time steps are three or more, the support system may perform the above-described fusion process on the two earliest time steps, then on the third earliest time step and on each of the time steps from the third time step onward (e.g., time steps t+2, t+3, ...), and the above fusion process may be performed based on the capture data corresponding to that time step and the fusion data obtained for the time step one time step prior to that time step. As a result, one fusion data can be input to the object detection system.

[0100] Semantic segmentation can include machine learning models (e.g., neural networks) trained on types of segmented objects commonly found in a train environment for controlling the autonomous operation of trains. The machine learning model may be a U-net or other type of neural network.

[0101] The process of recognizing pixels of the same object across different images (a first image and a second image) using a Siamese network can be an example of a tracking process for object tracking. Alternatively, an object tracking algorithm may be used as the tracking process instead of using a Siamese network. Examples of object tracking algorithms include Kalman filters using the Hungarian algorithm, Gaussian mixture stochastic hypothesis density filters, and orthogonal matching tracking methods for identifying approximate regions of the same object in different images (frames).

[0102] The process of recognizing pixels and point clouds of the same object across different images (first image and second image) using a Siamese network can be an example of a tracking process for object tracking. Instead of using a Siamese network, a tracking process involving point cloud comparison may be applied. Examples of object tracking algorithms include Kalman filters using the Hungarian algorithm to identify approximate regions of the same object in different images, Gaussian mixture stochastic hypothesis density filters, and orthogonal matching tracking methods. An example of a point cloud comparison algorithm may be geometric pattern matching with bending invariance.

[0103] The priority of timesteps may be modified depending on the nature of the application. For example, if an object detection support system is used for purposes other than autonomous train control support, the priority of timesteps may be different. [Explanation of Symbols]

[0104] 100 Sensor Systems

Claims

1. An object detection support system, An interface device configured to receive sensor data, including data captured by a sensor unit mounted on the vehicle, A storage device in which sensor data received by the interface device is stored, A processor coupled to the interface device and the storage device, It is equipped with, The processor obtains from the sensor data a first captured data, which is data captured in a first time step, and a second captured data, which is data captured in a second time step. The second time step is the time step following the first time step among a plurality of consecutive time steps, The processor calculates a first feature, which is a feature related to the first captured data, and a second feature, which is a feature related to the second captured data. The processor determines whether the relationship between the first feature and the second feature satisfies a predetermined condition. If the result of the above determination is true, the processor shall A fusion process is performed which includes fusing the first captured data with the second captured data. The first captured data is fused with the second captured data to produce fused data, which is output to an object detection system that detects objects based on the data input to the function. Object detection support system.

2. The aforementioned sensor unit is a camera unit, The shooting direction of the camera unit is the same as the direction of travel of the vehicle. The first captured data is the first image, The second captured data is the second image, The fused data includes a fused image which is an image in which the second image is fused into a region of the first image corresponding to the second image. The fusion process includes stitching a portion of an object in the first image to the same object partially captured in the second image. The object detection support system according to claim 1.

3. The processor generates first processed data by applying processing using semantic segmentation to the first image or an enhanced version of the first image. The processor performs a tracking process, which is a process for object tracking, based on the first processing data, the first feature quantity, and the second feature quantity. If a repeating object is specified by the aforementioned tracking process, The processor applies an inpainting process to the repeating object, In the aforementioned fused image, there is an object to which inpainting processing has been applied. The repeating object is an object in the first image, and is an object or part of an object in a region of the first image other than the region corresponding to the second image. The object detection support system according to claim 2.

4. The processor applies enhancement processing to at least the first image among the first and second images. The object detection support system according to claim 3.

5. The sensor unit is a LiDAR unit configured to capture an object in the direction of travel of the vehicle, The first captured data is a first point cloud, The second captured data is a second point cloud, The fused data includes a point cloud obtained by reflecting the first point cloud into the second point cloud. The object detection support system according to claim 1.

6. The interface device receives data not only from the LiDAR unit but also from the camera unit mounted on the vehicle. The shooting direction of the camera unit is the same as the direction of travel of the vehicle. The data received from the camera unit includes a first image, which is an image captured in the first time step, and a second image, which is an image captured in the second time step. The second point cloud is a point cloud corresponding to an object partially captured in the second image, The first point cloud is a point cloud that is partially captured in the second image and corresponds to the same object or part of an object that is captured in the first image. The object detection support system according to claim 5.

7. The processor generates first processed data by applying processing using semantic segmentation to the first image or an enhanced version of the first image. The processor performs a tracking process, which is a process for object tracking, based on the first processing data, the image projection data, the second point cloud, and the second image. The image projection data includes parameters relating to the camera unit and data representing the relative position between the camera unit and the LiDAR unit. If a repeating object is specified by the aforementioned tracking process, The aforementioned processor deletes the point cloud corresponding to the repeated object, The fused data does not include the deleted point cloud. The repeating object is an object in the first image, and is an object or part of an object in a region of the first image other than the region corresponding to the second image. The object detection support system according to claim 6.

8. The fused data includes a fused image which is an image in which the second image is fused into a region of the first image corresponding to the second image. The fusion process includes stitching a portion of an object in the first image to the same object partially captured in the second image. The object detection support system according to claim 6.

9. The determination is based on the displacement of the vehicle between the first time step and the second time step. The aforementioned predetermined condition is that the second feature quantity is similar to the first feature quantity. The object detection support system according to claim 1.

10. An object detection support method, The computer obtains, from sensor data including data captured by a sensor unit mounted on the vehicle, first captured data which is data captured at a first time step and second captured data which is data captured at a second time step, The second time step is the time step following the first time step among a plurality of consecutive time steps. To obtain, The computer calculates a first feature, which is a feature of the first captured data, and a second feature, which is a feature of the second captured data. The computer determines whether the relationship between the first feature and the second feature satisfies a predetermined condition, If the result of the above determination is true, The computer performs a fusion process which includes fusing the first captured data with the second captured data. The computer outputs fused data, which includes data obtained by fusing the first captured data with the second captured data, to the object detection system, which is the function that detects objects based on the data input to the function. A method for assisting object detection, including the method described above.

11. A system for controlling the autonomous operation of trains, An object detection system configured to detect objects from input data, The object detection support system according to claim 1, configured to support object detection by the object detection system, A driving control system configured to control the autonomous driving of the train based on the results of object detection by the object detection system, It is equipped with, The object detection system performs object detection based on the fusion data output from the object detection support system, and outputs the results of the object detection to the driving control system. system.

Images