Information processing apparatus, information processing method, and computer-readable non-transitory storage medium

Abstract

[Problem] To improve robustness of an inference device. [Solution] Provided is an information processing device comprising: an inference unit that performs inference on the basis of input sensing data using an inference means that is generated by training using masked sensing data as training data; and an abnormality detection unit that detects an abnormality related to the sensor that acquires the sensing data. When an abnormality of the sensor acquiring the sensing data is detected, the abnormality detection unit masks the sensing data acquired by the sensor and inputs the masked sensing data into the reasoning unit. The reasoning unit performs reasoning based on the sensing data of the mask input by the abnormality detection unit. At least part of the training data is generated using a generative AI.

Description

Technical Field

[0001] This disclosure relates to information processing apparatus, information processing methods, and non-transitory computer-readable storage media. Background Technology

[0002] In recent years, techniques have been developed to improve the inference accuracy of inference engines generated through machine learning. For example, Patent Document 1 discloses a technique for identifying raindrops attached to a vehicle's windshield with high accuracy using learning data generated by using a transparent plate with raindrops attached.

[0003] Reference List

[0004] Patent documents

[0005] Patent Document 1: Japanese Patent Application Publication No. 2021-61524 Summary of the Invention

[0006] The technical problem to be solved by the present invention

[0007] However, the technology disclosed in Patent Document 1 is specifically designed for identifying raindrops, and the diversity of the generated learning data is limited.

[0008] Technical solutions to technical problems

[0009] According to one aspect of this disclosure, an information processing apparatus is provided, comprising: an inference unit configured to perform inference based on input sense data using an inferencer generated by learning sense data using masked sense data; and an anomaly detection unit configured to detect anomalies associated with a sensor that has acquired sense data, wherein when the anomaly detection unit detects an anomaly in a sensor that has acquired sense data, the anomaly detection unit masks the sense data acquired by the sensor and inputs the sense data to the inference unit, the inference unit performing inference based on the masked sense data input by the anomaly detection unit, and at least a portion of the learning data is generated using generative AI.

[0010] Furthermore, according to another aspect of this disclosure, an information processing method executed by a processor is provided, the information processing method comprising: performing inference based on input sense data using an inferencer generated by learning sense data using masked sense data as learning data; and detecting anomalies associated with a sensor that has acquired sense data, wherein detecting anomalies comprises: masking the sense data acquired by the sensor when an anomaly is detected in the sensor that has acquired sense data; and performing inference comprises: performing inference based on the masked sense data, and at least a portion of the learning data is generated using generative AI.

[0011] Furthermore, according to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided that stores a program enabling a computer to be used as an information processing apparatus, the information processing apparatus comprising: an inference unit configured to perform inference based on input sense data using an inferencer generated by learning from masked sense data as learning data; and an anomaly detection unit configured to detect anomalies associated with a sensor that has acquired sense data, wherein when the anomaly detection unit detects an anomaly in a sensor that has acquired sense data, the anomaly detection unit masks the sense data acquired by the sensor and inputs the sense data to the inference unit, the inference unit performing inference based on the masked sense data input by the anomaly detection unit, and at least a portion of the learning data is generated using generative AI. Attached Figure Description

[0012] Figure 1 This is a diagram illustrating an outline of learning according to embodiments of the present disclosure.

[0013] Figure 2 This is a diagram illustrating the general outline of reasoning using a learned inference engine according to the same implementation method.

[0014] Figure 3 This is a block diagram illustrating a functional configuration example of an information processing apparatus 10 according to the same embodiment.

[0015] Figure 4 This is a diagram illustrating an example of an architecture for estimating segmentation, depth, and objects using sensor fusion via RGB camera 312, radar 314, and LiDAR 316, according to the same implementation.

[0016] Figure 5 This is a diagram illustrating an example of feature quantity fusion according to the same implementation method.

[0017] Figure 6 This is a diagram used to illustrate BEV mapping using camera parameters according to the same implementation method.

[0018] Figure 7 This is a diagram used to illustrate BEV mapping using camera parameters according to the same implementation method.

[0019] Figure 8 This is a diagram illustrating the configuration of an image generation AI 230 according to the same implementation, which generates an image for use as learning data 141 based on an input image and a selection context 145.

[0020] Figure 9This is a diagram illustrating the configuration of an image generation AI 230 according to the same implementation, which generates an image for use as learning data 141 based on an input image and a selection context 145.

[0021] Figure 10 This is a diagram illustrating the configuration for generating images for use as learning data 141 based on input annotations and selection context 145 for AI 230 of image generation according to the same implementation.

[0022] Figure 11 This is a diagram illustrating the configuration of an image generation AI 230, according to the same implementation, to generate images for use as learning data 141 based on input annotations and selection context 145.

[0023] Figure 12 This is a diagram illustrating the configuration of an image generation AI 230, which generates images for use as learning data 141 based on input object movement parameters, according to the same implementation.

[0024] Figure 13 This is a diagram illustrating the configuration of an image generation AI 230, which generates images for use as learning data 141 based on input object movement parameters, according to the same implementation.

[0025] Figure 14 This is a diagram illustrating the configuration of an image generation AI 230, which generates images for use as learning data 141 based on input object movement parameters, according to the same implementation.

[0026] Figure 15 This is a diagram illustrating the configuration of an image generation AI 230, which generates images for use as learning data 141 based on input object movement parameters, according to the same implementation.

[0027] Figure 16 This is a diagram used to illustrate context selection based on reasoning result analysis according to the same implementation method.

[0028] Figure 17 This is a diagram used to illustrate context selection based on reasoning result analysis according to the same implementation method.

[0029] Figure 18 This is a diagram used to illustrate context selection based on reasoning result analysis according to the same implementation method.

[0030] Figure 19 This is a diagram used to illustrate context selection based on reasoning result analysis according to the same implementation method.

[0031] Figure 20This is a diagram used to illustrate context selection based on reasoning result analysis according to the same implementation method.

[0032] Figure 21 This is a diagram illustrating the configuration related to the generation of learning data 141 using a sensor for cleaning, according to the same implementation.

[0033] Figure 22 This is a flowchart illustrating an example of a process for generating learning data 141 using sensors according to the same implementation.

[0034] Figure 23 This is a diagram illustrating an example of sensor data acquisition using an FMCW LiDAR 317 and an RGB camera 312 according to the same implementation.

[0035] Figure 24 This is for explaining the situation according to the same implementation method. Figure 23 The diagram shows the information that the inferencer 210 expects to learn.

[0036] Figure 25 This is a diagram illustrating the velocity components obtained from Doppler information of each point cloud of the FMCW LiDAR 317 according to the same implementation.

[0037] Figure 26 This is a diagram illustrating the process of generating learning data 141 using the FMCW LiDAR 317 according to the same implementation.

[0038] Figure 27 This is a diagram illustrating an example of mobile body region segmentation according to the same implementation. Figure 28 This is a diagram used to illustrate the calculation of the motion region of a moving body according to the same implementation method.

[0039] Figure 29 This is a diagram illustrating the generation of learning data 141 based on the results of motion calculation of the moving body region and the results of self-motion estimation (motion information of the stationary region) according to the same implementation method.

[0040] Figure 30 This is a diagram illustrating the process of generating a large-scale pre-learning dataset according to the same implementation method.

[0041] Figure 31 This is a diagram used to illustrate anomaly detection based on inter-frame difference maps according to the same implementation method.

[0042] Figure 32 This is a diagram illustrating anomaly detection based on optical flow deviation according to the same implementation method.

[0043] Figure 33 This is a diagram illustrating the reliability of reasoning-based anomaly detection according to the same implementation method.

[0044] Figure 34 This is a diagram illustrating anomaly detection based on the comparison of accumulated sensing data at stable locations, according to the same implementation method.

[0045] Figure 35 This is a flowchart illustrating an example of a process for anomaly detection based on focus position according to the same implementation.

[0046] Figure 36 This is a flowchart illustrating another example of a process for anomaly detection based on focus position according to the same implementation.

[0047] Figure 37 This is a flowchart illustrating an example of an anomaly detection process using a predetermined pattern according to the same implementation.

[0048] Figure 38 This is a diagram illustrating an example of feature points detected from an image obtained by imaging a predetermined pattern, according to the same embodiment.

[0049] Figure 39 This is a diagram illustrating the process of comparative learning according to the same implementation method.

[0050] Figure 40 This is a diagram used to illustrate the calculation of contrast loss in the contrast learning phase P1 according to the same implementation method.

[0051] Figure 41 This is a diagram illustrating anomaly detection based on a reliability comparison of multiple sensors according to the same implementation method.

[0052] Figure 42 This is a diagram illustrating a configuration for implementing anomaly detection based on the integration of inference results derived from multiple sensors, according to the same implementation method.

[0053] Figure 43 This is a flowchart illustrating an example of an anomaly detection process based on the integration of inference results derived separately from multiple sensors, according to the same implementation method.

[0054] Figure 44 This is a flowchart illustrating an example of a notification control process based on self-check results according to the same implementation.

[0055] Figure 45 This is a diagram illustrating a notification example of a response method, etc., according to the same implementation method.

[0056] Figure 46 This is a flowchart illustrating an example of a prompt control process according to an inspection method of the same implementation.

[0057] Figure 47 This is a diagram illustrating a prompt example of an inspection method according to the same implementation.

[0058] Figure 48 This is a flowchart illustrating an example of a reliable prompt control process according to the same implementation.

[0059] Figure 49 This is a diagram illustrating a display example of the reliability of the reasoning results according to the same implementation method.

[0060] Figure 50 This is a block diagram illustrating an example of a hardware configuration of an information processing device 10 installed on a moving body (e.g., a vehicle) according to the same embodiment, which performs inference using an inferencer 210 and sensor anomaly detection using an anomaly detection model 220.

[0061] Figure 51 This is a block diagram illustrating a hardware configuration example of an information processing apparatus 10 according to the same embodiment, which performs tasks such as generating learning data 141 and learning using the learning data 141. Detailed Implementation

[0062] In the following, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Note that in this specification and the drawings, components having substantially the same functional configuration are indicated by the same reference numerals, and redundant descriptions will be omitted.

[0063] Furthermore, in this specification and accompanying drawings, letters, etc., may be added to the end of the reference numerals when distinguishing and describing multiple configurations of the same type. However, when it is not necessary to distinguish multiple configurations of the same type, the aforementioned letters, etc., may be omitted, and the same description may be made for all multiple configurations of the same type.

[0064] Note that the descriptions will be given in the following order.

[0065] 1. Implementation Method

[0066] 1.1. Overview

[0067] 1.2. Function Configuration Example

[0068] 1.3. Generation of learning data

[0069] 1.4. Anomaly Detection Model 220 Sensor Anomaly Detection Method

[0070] 1.5. Notification Control

[0071] 1.6. Variations

[0072] 2. Hardware Configuration Examples

[0073] 3. Conclusion

[0074] <1. Implementation Method>

[0075] <<1.1. Summary>>

[0076] Techniques have been developed to perform inference based on sensing data acquired by sensors. These techniques include a concept called sensor fusion, which performs inference by integrating information from multiple sensors.

[0077] In cases such as sensor fusion, where inference is performed using sensing data acquired from multiple sensors, the inference engine learns normally based on the assumption that all sensors are operating correctly.

[0078] However, in this scenario, if an anomaly occurs in any of the sensors, the inference accuracy is significantly reduced.

[0079] Note that sensor-related anomalies include various situations that may cause a decrease in inference accuracy using sensing data acquired by the corresponding sensor.

[0080] Examples of sensor-related anomalies include malfunctions, damage, and the adhesion of foreign objects (such as contaminants or raindrops).

[0081] Technical concepts for embodiments according to this disclosure have been conceived with reference to the above points, and these technical concepts are intended to improve robustness against sensor-related anomalies.

[0082] To achieve the above, the learning method according to embodiments of this disclosure is characterized by using masked sensing data as learning data. That is, in the learning method according to embodiments of this disclosure, learning is performed by reproducing anomalies occurring in certain sensors.

[0083] Here, an outline of embodiments of this disclosure will be described. For example, embodiments of this disclosure may be implemented in the following process.

[0084] (1) Randomly mask part or all of the sensing data acquired by each sensor, and use the masked sensing data as learning data to perform the learning of the inference engine.

[0085] (2) Generate a sensor anomaly detection model in advance.

[0086] (3) Input the sensing data acquired by each sensor into the sensor anomaly detection model described above, and mask the abnormal area.

[0087] (4) Notify the user that an anomaly has been detected in the sensor.

[0088] (5) Input the sensing data of the abnormal region in (3) by the anomaly detection model mask into the inferencer generated in (1) and perform inference.

[0089] Figure 1 This is a diagram illustrating an outline of learning according to embodiments of the present disclosure.

[0090] like Figure 1 As shown, in the learning process according to this embodiment, the sensing data is masked using a masking parameter, and the masked sensing data is used as learning data to perform the learning of the inference engine.

[0091] The mask parameters mentioned above can be variables that determine the shape, size, frequency of occurrence, and value (e.g., 0, 255, etc. in the case of RGB images of sensing data).

[0092] Based on the learning data described above, it is possible to perform learning that reproduces situations where anomalies occur in certain sensors.

[0093] Next, we will refer to Figure 2 Describe the reasoning summary using the inference engine generated through learning.

[0094] like Figure 2 As shown, the sensing data acquired by the sensor is input into the anomaly detection model 220.

[0095] The anomaly detection model 220 performs sensor anomaly detection on the input sensing data and masks the abnormal region of the sensing data when an anomaly is detected.

[0096] The sensing data of the abnormal region, masked by the anomaly detection model 220, is input to the reference... Figure 1 The inference engine 210 described in the learning process.

[0097] The inference engine 210 performs inference based on the sensed data of the anomalous region being masked, and outputs the inference result.

[0098] In addition, if the anomaly detection model 220 detects an anomaly, it executes sensor anomaly notification control and notifies the user that an anomaly has been detected in the sensor.

[0099] Based on the above configuration, highly robust inference against sensor anomalies can be achieved.

[0100] <<1.2. Function Configuration Examples>>

[0101] Next, an example of the functional configuration of the information processing apparatus 10 according to this embodiment will be described. Figure 3 This is a block diagram illustrating an example of the functional configuration of the information processing apparatus 10 according to this embodiment.

[0102] According to this embodiment, the information processing apparatus 10 performs at least any of the following processes: learning related to the inferencer 210; generating learning data to be used for learning the inferencer 210; inference using the generated inferencer 210; and anomaly detection using the anomaly detection model 220.

[0103] Each of the above processes can be performed by a single information processing device 10, or by multiple information processing devices 10 respectively.

[0104] like Figure 3 As shown, the information processing device 10 according to this embodiment includes, for example, a sensor unit 110, an input unit 120, an output unit 130, a storage unit 140, a control unit 150, a location information acquisition unit 160, and a communication unit 170.

[0105] (Sensor unit 110)

[0106] The sensor unit 110 according to this embodiment includes multiple sensors for acquiring sensing data. Examples of sensors included in the sensor unit 110 include an RGB camera 312, a radar 314, a light detection and ranging device (LiDAR) 316, an inertial measurement unit (IMU) 318, etc. Figure 50 As shown in the figures.

[0107] (Input Unit 120)

[0108] The input unit 120 according to this embodiment receives user operations. The input unit 120 includes, for example, a keyboard, mouse, microphone, touch panel, switch, etc.

[0109] (Output Unit 130)

[0110] The output unit 130 of this embodiment outputs various types of information. The output unit 130 includes, for example, a display and a speaker.

[0111] (Storage unit 140)

[0112] According to this embodiment, the storage unit 140 stores various information to be used by the information processing device 10.

[0113] (Control Unit 150)

[0114] According to this embodiment, the control unit 150 controls each configuration included in the information processing device 10.

[0115] Furthermore, the control unit 150 according to this embodiment performs at least any of the following processes: learning related to the inferencer 210; generating learning data to be used for learning the inferencer 210; inference using the generated inferencer 210; and anomaly detection using the anomaly detection model 220.

[0116] For example, the control unit 150 according to this embodiment can operate as an inference unit that performs inference based on input sensing data using an inferencer 210, which is generated by learning using masked sensing data as learning data.

[0117] Furthermore, for example, the control unit 150 according to this embodiment can be operated as an anomaly detection unit that detects anomalies related to the sensor that has acquired sensing data by using the anomaly detection model 220.

[0118] Furthermore, for example, the control unit 150 according to this embodiment can operate as the inference unit and the anomaly detection unit described above. In this case, when the control unit 150 according to this embodiment detects an anomaly in the sensor that has acquired sensing data by using the anomaly detection model 220, the control unit masks the sensing data acquired by the sensor and inputs the masked sensing data into the inferencer 210 to perform inference based on the masked sensing data.

[0119] (Location information acquisition unit 160)

[0120] The location information acquisition unit 160 of this embodiment acquires the location of the information processing device 10. Therefore, the location information acquisition unit 160 of this embodiment includes a Global Navigation Satellite System (GNSS) receiver 365, for example, such as... Figure 50 As shown in the figure.

[0121] (Communication Unit 170)

[0122] According to this embodiment, the communication unit 170 performs information communication with another device via a network 50. The network 50 includes, for example, the Internet, a local area network (LAN), or a public network.

[0123] The above describes a functional configuration example of the information processing apparatus 10 according to this embodiment. Note that the above references... Figure 3 The described functional configurations are merely examples, and the functional configurations of the information processing device 10 are not limited to such examples.

[0124] The functional configuration of the information processing device 10 according to this embodiment can be flexibly modified according to specifications, operation, etc.

[0125] Next, we will refer to Figures 4 to 7 An example of an architecture for implementing inference according to this embodiment is described.

[0126] Figure 4 This is a diagram illustrating an example of an architecture for estimating segmentation, depth, and objects using sensor fusion with RGB camera 312, radar 314, and LiDAR 316.

[0127] First, the estimation of panoramic segmentation using sensor fusion of RGB camera 312 and LiDAR 316 will be described.

[0128] The 3D point cloud acquired by LiDAR 316 first undergoes depth transformation. Depth transformation is the process of converting the 3D point cloud acquired by LiDAR 316 into a 2D depth image.

[0129] Next, feature extraction is performed from the depth image obtained as described above (depth image feature extraction).

[0130] In addition, feature extraction (RGB image feature extraction) is performed on the RGB image acquired by the RGB camera 312.

[0131] Depth image feature extraction and RGB image feature extraction can be performed using network configurations employing convolutional neural networks (CNNs), deformable CNNs, or transformers. Furthermore, a pixel decoder can be added after the CNN or transformer to aggregate features across multiple scales. Note that the networks used for depth image feature extraction and those used for RGB image feature extraction can be prepared separately or the same.

[0132] Next, feature fusion is performed to combine depth image features with RGB image features.

[0133] Figure 5 This is a diagram illustrating an example of feature quantity fusion according to this embodiment.

[0134] exist Figure 5 In the example shown, vectors representing weather features, illuminance features, scene features, etc., are input to the prompt encoder.

[0135] The prompt encoder comprises a multilayer perceptron (MLP) with multiple layers and repeatedly performs the process of embedding the input vector into a high dimension and then transforming the vector into a one-hot vector.

[0136] Next, cross-attention is performed using the query (Q) obtained by multiplying the one-hot vector with the channels of the depth image feature quantity and using the RGB image feature quantity as the value (V) and key (K). The fused feature quantity is obtained by concatenating the result of the cross-attention with the original RGB image feature quantity.

[0137] Based on the processing described above, weather features, illumination features, and scene features can be used to optimize the capture of depth features.

[0138] Note that in Figure 5 The queries (Q), values ​​(V), and keys (K) shown are merely examples and can be flexibly modified.

[0139] Refer again Figure 4 Continue the description.

[0140] The fused feature values ​​obtained as described above, along with the query generated based on the fused feature values, are input into the segmentation transformer decoder. The segmentation transformer decoder is a panoramic segmentation decoder that performs segmentation of image region masks of the same category without distinguishing between background and foreground.

[0141] The processing result of the segment transformer decoder is post-processed and used as the result of semantic segmentation to obtain the result of instance segmentation.

[0142] Note that the post-processing described above includes the process of distinguishing each mask of a countable object (person, car, bicycle, etc.) from the results of panorama segmentation as a separate instance.

[0143] Furthermore, the post-processing described above includes grouping pixels of the same category from the results of panoramic segmentation and outputting the classification results of the categories as images on a pixel-by-pixel basis.

[0144] Next, we will describe the estimation of depth, object frame, and velocity using sensor fusion from RGB camera 312, radar 314, and LiDAR 316.

[0145] First, depth estimation is performed based on the extracted depth image features, RGB image features, and camera parameters.

[0146] The aforementioned camera parameters are various parameters used to project the point cloud of LiDAR 316 onto an RGB image, and include, for example, information such as the pose (rotation matrix) and position (translation vector) of the RGB camera 312, the focal length, and the center position of the lens.

[0147] Next, a bird's-eye view (BEV) mapping is performed based on the depth estimated as described above.

[0148] However, various preprocessing and radar feature extraction are performed on the point cloud acquired by radar 314, and BEV mapping is performed based on the extracted radar features.

[0149] Subsequently, BEV feature fusion is performed on the results based on BEV mapping (depth-based) and the results based on BEV mapping (radar feature quantity), and the obtained fused feature quantity is input into the object detection decoder to obtain the estimation results of object box and velocity.

[0150] According to the reference Figure 4 and Figure 5 The architecture described above improves the estimation accuracy for each task compared to existing technologies.

[0151] However, Figure 4 and Figure 5 The architecture shown is merely an example, and the architecture according to this implementation can be flexibly modified.

[0152] For example, such as Figure 6 As shown, the BEV mapping according to this embodiment can be achieved by mapping two-dimensional feature quantities to BEV using the estimated depth and camera parameters. In this case, by explicitly calculating the depth within the model, it is expected to improve the object detection accuracy.

[0153] Furthermore, in the BEV mapping according to this embodiment, such as Figure 7 As shown, information obtained by radar 314 can also be used.

[0154] In this case, BEV mapping is performed using information about the xyz values ​​acquired by radar 314 and the rotation and translation matrix of radar 314. At this point, in the absence of metadata such as velocity information, only occupancy information is included. Therefore, the RGB image features are concatenated with the result of dimension 1, which indicates the presence of detection points. However, if metadata exists, the result corresponding to the dimension of the metadata is concatenated with the RGB image features.

[0155] By using information obtained from radar 314 for BEV mapping, it is expected that the accuracy of velocity estimation will be improved.

[0156] Furthermore, in the BEV mapping according to this embodiment, extracted contextual feature quantities, such as... Figure 6 and Figure 7 As shown in the image.

[0157] <<1.3. Generation of Learning Data>>

[0158] Next, a method for generating learning data according to this embodiment will be described.

[0159] At least a portion of the learning data according to this embodiment can be generated using generative artificial intelligence (AI).

[0160] The generation of learning data 141 for AI 230 using images will be described below.

[0161] Figure 8 This is an illustration of the configuration for generating an image for use as learning data 141 based on an input image and a selection context 145 according to this embodiment of the image generation AI 230.

[0162] exist Figure 8 In the example shown, the image (real-time motion picture, computer graphics (CG)) of the pre-prepared annotated image data 142 is input into the image generation AI 230.

[0163] In addition, for example, a selection context 145 randomly selected from context data 144 is input into the image generation AI 230 along with the aforementioned image.

[0164] Context data 144 includes information on changes in each of various items, such as weather conditions, time of day, sensor anomalies, solar radiation, and location.

[0165] Examples of changes in weather conditions include sunny, rainy, cloudy, snowy, lightning, storm, fog, etc.

[0166] In addition, examples of changes in time of day include morning, daytime, evening, night, early morning, etc.

[0167] In addition, examples of abnormal changes in sensors include water droplet adhesion, mud adhesion, internal malfunction, and damage.

[0168] In addition, examples of changes in solar radiation include front lighting, back lighting, and afternoon sunlight.

[0169] In addition, examples of variations of location include intersections, motor vehicle lanes only, highways, railway crossings, agricultural roads, and unpaved roads.

[0170] The control unit 150 selects an element from the aforementioned changes for each item included in the context data 144.

[0171] For example, the control unit 150 selects a set of conditions such as "weather condition: sunny", "time of day: evening", "sensor anomaly: mud adhesion", "solar radiation: afternoon sunlight", and "location: intersection" from the context data 144, and inputs the selected set into the image generation AI 230 as the selection context 145.

[0172] Image generation AI 230 generates images based on the input selection context 145 and the image.

[0173] The images generated by the image generation AI 230 are used as learning data 141 in association with the selection context 145 and the annotations extracted from the annotated image data 142.

[0174] According to the method described above, a large amount of learning data reflecting various environments, sensor anomalies, etc. can be generated in a short time.

[0175] Furthermore, by using a wide variety of large amounts of learning data 141 to perform learning for the inference engine 210, the robustness of the inference engine 210 can be improved.

[0176] Note that developers can choose the context.

[0177] For example, Figure 9 The configuration for determining the selection context 145 is shown by performing speech recognition on the developer's speech.

[0178] Such a configuration is particularly effective when it is desired to prepare learning data with features specific to a particular context.

[0179] Furthermore, the image generation AI 230 according to this embodiment can generate images to be used as learning data 141 based on the selected context 145 and annotations.

[0180] Figure 10 This is a diagram illustrating the configuration of the image generation AI 230 according to this embodiment to generate images for use as learning data 141 based on input annotations and selection context 145.

[0181] exist Figure 10 In the example shown, annotations are input into the image generation AI 230 from the pre-prepared annotation mask data 143.

[0182] Here, as in the semantic segmentation estimation results, the annotation mask data 143 can be, for example, data with segmentation information and annotations.

[0183] In this case, the image generation AI 230 generates an image based on the input annotations and selection context 145.

[0184] The image generated by the image generation AI 230 is used as learning data 141 in association with the selection context 145 and the annotations extracted from the annotation mask data 143.

[0185] Note that, similarly, when annotation mask data 143 is used to generate learning data 141, developers can choose the context, such as... Figure 11 As shown.

[0186] Furthermore, the image generation AI 230 according to this embodiment can generate an image to be used as learning data 141 based on the input object movement parameters.

[0187] Figures 12 to 15 This is used to illustrate the configuration of the image generation AI 230 according to this embodiment to generate an image to be used as learning data 141 based on the input object movement parameters.

[0188] Notice, Figure 12 The annotated image data 142 is shown for generating learning data 141, and the context selection can be configured to be performed solely by the control unit 150.

[0189] also, Figure 13 The annotated image data 142 is shown to be used to generate learning data 141, and the context selection can also be configured by the developer.

[0190] also, Figure 14 The diagram shows the configuration where annotation mask data 143 is used to generate learning data 141, and the context selection can be performed solely by the control unit 150.

[0191] also, Figure 15 The annotation mask data 143 is shown for generating learning data 141, and the context selection can also be configured by the developer.

[0192] exist Figures 12 to 15 In the example shown, in addition to annotations and images, the object movement parameter is also input into the image generation AI 230.

[0193] Here, the object movement parameter can be a parameter used to specify the amount and direction of movement of objects included in the image.

[0194] Image generation AI 230 generates images of object movement based on input object movement parameters.

[0195] Based on this method, it is easy to generate learning data with various variations in the position of objects.141

[0196] Next, we will refer to Figures 16 to 19 The selection of context for describing the analysis of the inference results based on inferencer 210.

[0197] Notice, Figure 16 The annotated image data 142 is shown for generating learning data 141, and the context selection can be configured to be performed solely by the control unit 150.

[0198] also, Figure 17 The annotated image data 142 is shown to be used to generate learning data 141, and the context selection can also be configured by the developer.

[0199] also, Figure 18 The diagram shows the configuration where annotation mask data 143 is used to generate learning data 141, and the context selection can be performed solely by the control unit 150.

[0200] also, Figure 19 The annotation mask data 143 is shown for generating learning data 141, and the context selection can also be configured by the developer.

[0201] exist Figures 16 to 19 In the example shown, the generated learning data 141 is used to perform inference learning to generate inference 210.

[0202] In addition, test data 146 is extracted from the generated learning data 141 and the test data is input into the inference engine 210.

[0203] Inferencer 210 performs inference based on the input test data and outputs the inference results.

[0204] The control unit 150 analyzes the reasoning results of the inference engine 210 and uses the analysis results for context selection.

[0205] Figure 20 This is a diagram used to illustrate context selection based on reasoning result analysis according to this embodiment.

[0206] The control unit 150 first performs a quantitative evaluation of each context based on the inference results of the inference engine 210. At this time, if the inference result is semantic segmentation, an evaluation index such as mean IoU can be used, and if the inference result is depth, an evaluation index such as RMSE, MAE, or REL can be used.

[0207] Next, the control unit 150 extracts difficult scenarios based on the quantitative evaluation results. For example, the control unit 150 can extract contexts with inference accuracy below a set threshold and contexts with estimation accuracy significantly lower than the average accuracy, and can define scenarios related to the extracted contexts as difficult scenarios.

[0208] In addition, the control unit 150 actively performs context selection related to the extracted difficult scenario.

[0209] Based on the processing described above, by generating a large amount of learning data 141 with context related to difficult scenarios and performing learning using the learning data 141, the robustness of the inferencer 210 can be effectively improved.

[0210] In addition, the control unit 150 can visualize the quantitative evaluation results by using tables, charts, etc., so as to present the quantitative evaluation results to the developers.

[0211] The generation of learning data 141 for AI 230 using images according to this embodiment has been described above.

[0212] Next, the generation of learning data 141 using sensors according to this embodiment will be described.

[0213] Figure 21 This is a diagram illustrating the configuration related to the generation of learning data 141 using sensors for cleaning according to this embodiment.

[0214] exist Figure 21 In the example shown, the control unit 150 sends an imaging trigger and a cleaning trigger to the sensor unit 110 based on the input sensor cleaning signal, and causes the sensor to perform sensor cleaning and acquire sensing data before and after sensor cleaning.

[0215] The sensor sensing data before and after cleaning, obtained through the above control, are registered as learning data 141.

[0216] Next, we will refer to Figure 22 The process of generating learning data 141 using sensors according to this embodiment is described in detail. Figure 22 This is a flowchart illustrating an example of a process for generating learning data 141 using sensors according to this embodiment.

[0217] exist Figure 22 In the example shown, first, the control unit 150 receives a sensor cleaning signal (S101).

[0218] Based on the center cleaning signal received in step S101, the control unit 150 sends an imaging trigger to the sensor unit 110 (S102).

[0219] In step S102, the sensor unit 110, which receives the imaging trigger, acquires the sensing data before cleaning based on the imaging trigger (S103).

[0220] The control unit 150 registers the sensing data obtained in step S103 before cleaning as learning data 141 (S104).

[0221] Next, the control unit 150 sends a cleaning trigger to the sensor unit 110 (S105).

[0222] In step S105, the sensor unit 110 receives a cleaning trigger and performs sensor cleaning based on the cleaning trigger (S106).

[0223] Next, the control unit 150 sends an imaging trigger to the sensor unit 110 again (S107).

[0224] In step S107, the sensor unit 110, which receives the imaging trigger, acquires the cleaned sensing data based on the imaging trigger (S108).

[0225] The control unit 150 registers the cleaned sensing data acquired in step S108 as learning data 141 (S109).

[0226] Based on the above-described method for generating learning data, a large amount of learning data can be generated with high precision and various variations according to the contamination status of the sensor.

[0227] Next, the generation of learning data 141 using a frequency modulated continuous wave (FMCW) LiDAR 317 according to this embodiment will be described.

[0228] The FMCW LiDAR 317 is a LiDAR capable of detecting the distance and velocity of objects.

[0229] Figure 23 This is a diagram illustrating an example of sensing data acquisition using an FMCW LiDAR 317 and an RGB camera 312 according to this embodiment.

[0230] exist Figure 23 In the example shown, the RGB camera 312 captures RGB images of objects O1 and O2 as subjects, and the FMCW LiDAR 317 can detect the distance and velocity of objects O1 and O2.

[0231] Figure 24 It is used to explain about Figure 23 The diagram shows the information that the inferencer 210 expects to learn.

[0232] like Figure 24 As shown by the dashed arrow in the image, the inferencer 210 needs flow (motion) information in each pixel of the image to learn.

[0233] However, as Figure 25 As shown, the velocity component obtained from the Doppler information of each point cloud of the FMCW LiDAR 317 is the component obtained by projecting the actual relative motion vector (indicated by the white arrow) of the object relative to the direction of the beam (indicated by the dashed line), and therefore, the velocity component is different from the actual relative motion.

[0234] Therefore, when using FMCW LiDAR 317 to generate learning data 141, it is necessary to estimate the actual relative motion by using information about multiple points.

[0235] Figure 26 This is a diagram illustrating the process of generating learning data 141 using the FMCW LiDAR 317 according to this embodiment.

[0236] First, the control unit 150 performs self-motion estimation based on information acquired by the FMCW LiDAR 317, RGB camera 312, IMU 318, GNSS receiver 365, etc.

[0237] Next, the control unit 150 performs moving volume region extraction by canceling the self-motion component obtained from the self-motion estimation above, and extracting only the moving region from the point cloud of the motion information of the FMCW LiDAR 317.

[0238] Next, the control unit 150 groups the aggregates of point clouds that have motion and similar distances and motion, and performs moving body region segmentation on the image to segment the moving regions of objects including each group.

[0239] Figure 27 This is a diagram illustrating an example of mobile body region segmentation according to this embodiment. Figure 27 In the image, the moving point cloud is represented by the shadowed circles.

[0240] Next, the control unit 150 performs motion calculation of the moving body region based on the results of moving body region extraction and moving body region segmentation. The motion calculation of the moving body region is a process of calculating the motion of the object by fitting motion components of multiple points obtained from the FMCW LiDAR 317 in each motion segment.

[0241] Figure 28 This is a diagram used to illustrate the motion calculation of the moving body region according to this embodiment.

[0242] Note that in Figure 28 In the diagram, the actual motion vector of object O is indicated by a white arrow, and the motion components in each beam direction obtained by the FMCW LiDAR 317 are indicated by black arrows.

[0243] Figure 28 The upper part shows the actual motion vector of object O when it moves upward and the motion components in each beam direction obtained by the FMCW LiDAR317 [1] to [3].

[0244] Figure 28The middle section shows the actual motion vector of object O as it moves in the direction of FMCW LiDAR 317, as well as the motion components in each beam direction obtained by FMCW LiDAR 317 [1] to [3].

[0245] Figure 28 The lower part shows the actual motion vector of object O as it moves downward, as well as the motion components in each beam direction obtained by the FMCW LiDAR317 [1] to [3].

[0246] In addition, on the right side of each section, a diagram showing the superposition of motion components[1] to [3] in each beam direction is shown.

[0247] like Figure 28 As shown, the motion components obtained by the FMCW LiDAR 317 change according to the angle between the beam direction and the actual motion vector of object O. Therefore, as long as three or more ranging points are obtained, the motion vector of object O can be calculated through fitting.

[0248] Finally, the control unit 150 performs projection onto the image based on the results of motion calculation of the moving body region and the results of self-motion estimation (motion information of the stationary region) to generate learning data 141.

[0249] Figure 29 This is a diagram illustrating how learning data 141 is generated based on the results of the moving body region motion calculation and the results of the self-motion estimation (motion information of the stationary region) according to this embodiment.

[0250] Figure 29 The upper left shows the motion vector of each moving body region obtained through the motion calculation of the moving body region, and the lower left shows the motion vector of the static region calculated based on the self-motion information and the distance data of the point cloud of FMCW LiDAR 317.

[0251] The control unit 150 generates, by performing coordinate transformation to the coordinate system of the RGB camera 312 and synthesizing the two motion vectors mentioned above, a vector as shown in the figure. Figure 29 The learning data 141 is shown on the right.

[0252] According to the method described above, a large amount of learning data can be generated at low cost.141

[0253] Next, the generation of a large-scale pre-learning dataset according to this embodiment will be described.

[0254] Recent research has shown that generalization performance across various tasks can be improved by pre-learning the backbone (feature extractor) using large-scale datasets that include pairs of similar images and pairs of images and text (descriptive text) (image-image contrastive learning and image-text contrastive learning).

[0255] However, it is said that a very large amount of data (millions to billions of records) is required to perform pre-learning using the methods described above, and preparing a large number of accurate image and text pairs has become a problem.

[0256] Therefore, this embodiment proposes a method to generate a large number of image and text pairs by organizing imaging data and map information and using AI to perform explanatory text generation, etc.

[0257] Figure 30 This is a diagram illustrating the process of generating a large-scale pre-learning dataset according to this embodiment.

[0258] exist Figure 30 In the example shown, the information processing device 10 is installed on a data collection vehicle and acquires a large amount of data through long-term driving.

[0259] First, image acquisition and GNSS data acquisition are performed based on an imaging trigger issued by the control unit 150. Image acquisition and GNSS data acquisition are synchronized via a synchronization signal.

[0260] Explanatory text 148 is generated from the acquired image by using AI, optical character recognition (OCR), and other methods.

[0261] Furthermore, based on the acquired GNSS data, explanatory text 148 can be generated based on map information extracted from map data 147.

[0262] Based on the method for generating explanatory text 148 as described above, it is possible to easily obtain a large amount of text related to place names, addresses, signs, markers, and other various scenarios.

[0263] The generated descriptive text 148 is registered in the pre-learning dataset 149 in association with the acquired image.

[0264] In addition, the explanatory text 148 is input into the image generation AI 230 and is registered in the pre-learning dataset 149 in association with the image generated by the image generation AI 230.

[0265] Furthermore, the explanatory text 148 can be used to search the Internet or large-scale datasets, and can be associated with images obtained as search results and registered in the pre-learning dataset 149.

[0266] Using the method described above, it is possible to create large and accurate datasets of images and text at low cost.

[0267] <<1.4. Anomaly Detection Model 220 Sensor Anomaly Detection Method>>

[0268] Next, the sensor anomaly detection method of the anomaly detection model 220 according to this embodiment will be described in detail.

[0269] For example, the anomaly detection model 220 according to this embodiment can detect anomalies related to the sensor that acquired the sensing data based on the changes in the acquired sensing data in the time direction.

[0270] For example, when the sensed data is an image, the anomaly detection model 220 can detect sensor-related anomalies based on image features in the time direction.

[0271] First, refer to Figure 31 This embodiment describes an anomaly detection based on inter-frame difference maps. The anomaly detection method is based on the prediction that if, despite the movement of a moving object such as a vehicle equipped with the information processing device 10, there are areas in the image that hardly change over time, then the probability of an anomaly occurring in the sensor that acquired the image is high.

[0272] First, image acquisition and vehicle speed sensor data acquisition are performed based on an imaging trigger issued by the control unit 150. A synchronization signal synchronizes the image acquisition and vehicle speed sensor data acquisition.

[0273] The acquired images are accumulated in the frame memory.

[0274] The control unit 150 calculates a difference map (diff) based on the newly acquired image and the image from the previous frame. The control unit 150 can use sum of absolute differences (SAD), zero-mean normalized cross-correlation (ZNCC), or AI-based matching to calculate the difference map (diff) for each pixel.

[0275] Subsequently, the control unit 150 calculates the anomaly map E based on the calculated difference map diff and the speed v obtained from the vehicle speed sensor data.

[0276] For example, the upper limit of the difference parameter is diff 最大 And the upper limit of the speed parameter is v. 最大 In the case of an anomaly, the control unit 150 uses the following formula (1) to calculate the anomaly diagram E.

[0277] [Mathematical Expression 1]

[0278] An anomaly graph E=1 indicates a high probability of an anomaly in the sensor. Conversely, an anomaly graph E=0 indicates a high probability of no anomaly in the sensor.

[0279] Based on the anomaly detection method based on the differences between frames as described above, it is possible to detect anomalies related to the sensor that acquired the image with high accuracy.

[0280] Next, we will refer to Figure 32 This embodiment describes an anomaly detection method based on optical flow deviation. The anomaly detection method is based on the prediction that if an anomaly exists in a sensor that has acquired an image, the temporal average of the optical flow is likely to be small due to the small temporal variation on the image.

[0281] First, image acquisition is performed based on an imaging trigger issued by the control unit 150.

[0282] The acquired images are accumulated in the frame memory.

[0283] Next, the control unit 150 calculates the optical flow using either a model-based method (such as the Lucas-Kanade method) or an AI-based method.

[0284] The calculated optical flow is accumulated in the frame memory.

[0285] The control unit 150 calculates the anomaly map E(x) based on the accumulated optical flow.

[0286] For example, the optical flow at time t is F t (x) and the threshold used for anomaly detection is F. th In the case of an anomaly, the control unit 150 uses the following formula (2) to calculate the anomaly map E(x).

[0287] [Mathematical Expression 2]

[0288] Based on the optical flow-based anomaly detection described above, anomalies associated with the sensor that acquired the image can be detected with high precision.

[0289] Next, we will refer to Figure 33 This embodiment describes an anomaly detection method based on inference reliability. The anomaly detection method is based on the prediction that if there are regions across multiple frames where the inference reliability remains consistently low, then there is a high probability that an anomaly has occurred in the sensor.

[0290] First, the inferencer 210 performs inference based on the input sensed data and outputs the inference result and the reliability associated with the inference result.

[0291] The reliability output by inferencer 210 is accumulated in frame memory.

[0292] The control unit 150 calculates the anomaly graph E(x) based on the accumulated reliability.

[0293] For example, the optical flow at time t is C t (x) and the threshold used for anomaly detection is C. th In the case of an anomaly, the control unit 150 uses the following formula (3) to calculate the anomaly map E(x).

[0294] [Mathematical Expression 3]

[0295] Based on the reliability-based anomaly detection described above, anomalies related to sensors that have acquired sensing data can be detected with high accuracy.

[0296] Next, we will refer to Figure 34 This embodiment describes anomaly detection based on comparison of accumulated sensing data at stable locations.

[0297] This anomaly detection method is based, for example, on the prediction that accumulated sensing data in stable locations such as parking lots at home or workplaces change very little over long periods of time.

[0298] First, when a mobile vehicle equipped with the information processing device 10 is parked at a predetermined location such as a parking lot at home or workplace, an imaging trigger is issued when the user notifies the system or when the control unit 150 detects parking at the predetermined location.

[0299] The control unit 150 normalizes the sensing data acquired based on the imaging trigger using normalization parameter 242. At this time, the control unit 150 can perform normalization using a min-max method, a statistical method such as Z-score normalization, or AI.

[0300] Next, the control unit 150 calculates the difference map diff based on the normalized sensing data and the previously accumulated normalized data 241. The control unit 150 can use SAD, ZNCC, or AI-based matching to calculate the difference map diff for each pixel.

[0301] Subsequently, the control unit 150 calculates the anomaly map E based on the calculated difference map diff.

[0302] For example, the upper limit of the difference parameter is diff 最大 In the case of an anomaly, the control unit 150 uses the following formula (4) to calculate the anomaly diagram E.

[0303] [Mathematical Expression 4]

[0304] Next, the control unit 150 performs anomaly determination based on the calculated anomaly map E and the normalized sensing data.

[0305] If all pixels in the abnormal image E are equal to or less than the threshold, the control unit 150 can determine that there is no abnormality and add the normalized sensing data to the normalized data 241.

[0306] Based on the anomaly detection method described above, which is based on the comparison of accumulated sensing data at stable locations, it is possible to detect anomalies related to the sensors that have acquired the sensing data with high accuracy.

[0307] The above describes anomaly detection based on image feature quantities in the time direction according to this embodiment.

[0308] However, the anomaly detection model 220 according to this embodiment can also detect anomalies based on image features in the spatial direction.

[0309] For example, the anomaly detection model 220 according to this embodiment can detect anomalies related to the sensor that acquired the image based on the focus position of the image.

[0310] Figure 35 This is a flowchart illustrating an example of an anomaly detection process based on focus position according to this embodiment. Figure 35 The example shown is based on the prediction that if the focus points in an image are very close, there is a high probability that anomalies such as contamination will occur in the sensor.

[0311] exist Figure 35 In the example shown, the control unit 150 first detects the focus position of each pixel or each small region comprising multiple pixels (step S201). At this time, the control unit 150 can detect the focus position in all pixels or small regions, or it can select some discretely distributed pixels or small regions to detect the focus position.

[0312] Next, the control unit 150 will detect pixels or small areas that exceed a certain threshold or fail to specify a focus position as abnormal areas (S202).

[0313] Subsequently, the control unit 150 notifies the user of the detected abnormal area (S203).

[0314] Figure 36 This is a flowchart illustrating another example of the process for anomaly detection based on focus position according to this embodiment. Figure 36 The example shown is based on the prediction that if the focus position does not change in multiple images with different scenes, there is a high probability that anomalies such as contamination will occur in the sensor.

[0315] exist Figure 36 In the example shown, firstly, the control unit 150 detects the focus position of each pixel or each small region comprising multiple pixels (S301) and records the detected focus position (S302).

[0316] Control unit 150 repeats steps S301 and S302 until the number of images recorded at the focus position exceeds the threshold (Yes in step S303).

[0317] When the number of images recorded at the focus position exceeds the threshold (S303: Yes), the control unit 150 calculates the variance of the focus position for each recorded point (pixel or small area) (S304).

[0318] Next, the control unit 150 detects the recording points with variance less than the threshold as abnormal regions (S305).

[0319] In addition, the control unit 150 notifies the user of the detected abnormal area (S306).

[0320] The above describes anomaly detection based on focus position. Note that the start timing of processing can be any timing, as long as the sensor can specify the focus position at high speed, such as algorithms that examine the image plane phase difference. However, when specifying the focus position using time-consuming algorithms (such as algorithms that specify the focus position by examining contrast), it is desirable that the scene is stationary and the vehicle is also stationary. Therefore, in this case, it is desirable to start processing at a timing after stopping or while waiting at a traffic light.

[0321] Next, we will refer to Figure 37 and Figure 38 Anomaly detection using a predetermined pattern according to this embodiment is described. In this detection method, a predetermined pattern is printed on the storage location of a sensor or on a cover, and anomaly detection is performed based on feature points detected from an image obtained by imaging the predetermined pattern.

[0322] The predetermined pattern can be, for example, a pattern that facilitates feature point detection, such as a grid pattern or a polka dot pattern, or it can be a complex pattern such as a QR code (registered trademark).

[0323] Figure 37 This is a flowchart illustrating an example of an anomaly detection process using a predetermined pattern according to this embodiment.

[0324] exist Figure 37 In the example shown, the control unit 150 first detects feature points from the image that forms the predetermined pattern (S401).

[0325] Figure 38 This is a diagram illustrating an example of feature points detected from an image obtained by imaging a predetermined pattern.

[0326] Figure 38 The top left side shows an example of a feature point detected when there is no abnormal area in the sensor (indicated by a black cross), and the top right side shows an example of a feature point detected when there is an abnormal area in the sensor.

[0327] Next, the control unit 150 predicts the location of the feature points that should be detected (S402). The control unit 150 can predict the feature points that should be detected by using information about the predetermined pattern, linearity, etc.

[0328] exist Figure 38 On the lower left, instances of the predicted feature points that should have been detected are indicated by white cross markers.

[0329] Next, the control unit 150 designates the abnormal region based on the location of the detected feature point (detection location) and the predicted location of the feature point that should have been detected (predicted location) (S403).

[0330] For example, if the detected location and the predicted location are separated from each other by more than a threshold, the control unit 150 determines that there is an anomaly in the predicted location. The control unit 150 can mask a square region with sides of σ pixels, centered on the predicted location to be determined as an anomaly, or it can mask the region with a circle of radius r pixels. Alternatively, the control unit 150 can divide the image into multiple grid-like regions, and if a predicted location exists in the divided regions, the divided regions can be masked as an anomaly region.

[0331] Figure 38 The lower right side shows an example of an image where the abnormal area is masked.

[0332] The control unit 150 notifies the user of the detected abnormal area (S404).

[0333] The above describes anomaly detection based on image feature quantities in spatial direction according to this embodiment. Based on the anomaly detection described above, anomalies related to the sensor that acquired the sensing data can be detected with high accuracy.

[0334] Next, the anomaly detection based on contrastive learning according to this embodiment will be described.

[0335] Figure 39 This is a diagram illustrating the comparative learning process according to this embodiment.

[0336] like Figure 39As shown, the anomaly detection based on contrastive learning according to this embodiment is roughly divided into a contrastive learning phase P1, an anomaly detection learning phase P2, and an anomaly detection phase P3.

[0337] In the contrastive learning phase P1, after acquiring positive and negative image patches, contrastive learning is performed using the acquired image patches.

[0338] Positive image blocks are blocks of images acquired by a normal sensor, while negative image blocks are blocks of images acquired when an anomaly occurs in the sensor. Note that a positive image can be extracted from a negative image.

[0339] In the anomaly detection learning phase P2, the encoder, which has undergone contrastive learning, extracts the feature values ​​of the learning samples and learns a classifier based on the extracted feature values. This classifier is used to determine whether the region is an anomaly region.

[0340] In the subsequent anomaly detection phase P3, image acquisition is performed, feature extraction is carried out using an encoder that has undergone contrastive learning, an anomaly detection classifier is executed, anomaly determination is performed based on the classifier's execution results, and notification is sent based on the anomaly determination results.

[0341] Figure 40 This is a diagram illustrating the calculation of contrast loss in the contrast learning phase P1 according to this embodiment.

[0342] like Figure 40 As shown, enhancements are applied to both positive and negative image patches. Examples of enhancement methods include color dithering, cropping, blurring, rotation, random erasing, etc.

[0343] Subsequently, each image is input into the encoder, and the latent feature quantity h is obtained. Furthermore, the latent feature quantity h is input into the MLP, and the projected feature quantity z is obtained.

[0344] Next, based on the obtained projection feature z, the contrast loss P11 and P12 are calculated using the following formula (5).

[0345] [Mathematical Expression 5]

[0346] Z in formula (5) i and Z j This represents the projection feature quantity of the enhanced image from the positive image patch. Furthermore, Z in equation (5) k Z represents the projection feature quantity (k≠i) of the enhanced image derived from the negative image patch. i and Z j This indicates that it is correct, and Z i and Z kIt indicates a negative pair.

[0347] In addition, N in formula (5) represents the batch size, and Sim represents the cosine value of the feature (larger if similar, smaller if different).

[0348] In the contrastive learning according to this embodiment, learning is performed based on the contrastive loss obtained by calculating P11 through contrastive loss, so that the feature quantities of positive image patches are close to each other. Furthermore, learning is performed based on the contrastive loss obtained by calculating P12 through contrastive loss, so that the feature quantities of positive and negative image patches are different.

[0349] The above describes an anomaly detection based on contrastive learning according to this embodiment.

[0350] Next, we will refer to Figure 41 This paper describes anomaly detection based on reliability comparisons across multiple sensors. The anomaly detection method compares the reliability of estimates based on each of the multiple sensors and identifies regions where the reliability deviates as anomalies.

[0351] exist Figure 41 In the example shown, the control unit 150 performs 2D segmentation estimation and reliability estimation of the 2D segmentation estimation based on the image acquired by the RGB camera 312.

[0352] In addition, the control unit 150 performs 3D segmentation estimation and reliability estimation of 3D segmentation based on point cloud data acquired by LiDAR 316.

[0353] Next, the control unit 150 projects the reliability estimation result of the 3D segmentation estimation onto 2D and compares the reliability maps.

[0354] If a region deviating from the reliability is identified in the comparison of the reliability maps, the control unit 150 detects that region as an abnormal region.

[0355] Next, we will refer to Figure 42 and Figure 43 This embodiment describes anomaly detection based on a comparison of inference results derived from multiple sensors.

[0356] Figure 42 This is a diagram illustrating a configuration for anomaly detection based on the integration of inference results derived from multiple sensors, according to this embodiment. Furthermore, Figure 43 This is a flowchart illustrating an example of an anomaly detection process based on the integration of inference results derived separately from multiple sensors, according to this embodiment.

[0357] The inference engine 210 performs multiple inferences based on the sensing data acquired by the various sensors included in the sensor unit 110.

[0358] For example, inferencer 210 can perform semantic segmentation based on sensing data acquired by a specific sensor, and can perform instance segmentation based on sensing data acquired by another sensor.

[0359] The control unit 150 projects the results of semantic segmentation, instance segmentation, etc. onto the map based on multiple inference results of the inferencer 210 and the results of self-position estimation (S501).

[0360] At this time, the control unit 150 can integrate the inference results projected onto the map, and accumulate and average the inference results derived from multiple sensors respectively.

[0361] Furthermore, the control unit 150 can arrange the line of sight on a sphere centered on the target, or it can divide the line of sight onto a cylindrical surface. When the information processing device 10 is mounted on a moving body, it is assumed that the direction of the desired line of sight is on a plane. Even if the direction of the line of sight is limited, the impact on the results is minor, and the efficiency of storage and processing is improved by employing a cylindrical surface.

[0362] Furthermore, the control unit 150 can integrate the results from all viewpoints of a single sensor, or it can integrate the results from multiple sensors. For example, the control unit 150 can integrate the results of semantic segmentation, etc., for each sensor and each viewpoint, and when determining the results of semantic segmentation, etc., of the map, it can integrate the results of each sensor and each viewpoint. Additionally, the map can be a point cloud, an occupancy map, or a grid.

[0363] The anomaly detection model 220 will project the map based on the reasoning results as described above onto the sensor surface of the target sensor (S502).

[0364] Next, the anomaly detection model 220 compares the inference results stored in the map with the inference results about the target sensor (S503).

[0365] Anomaly detection model 220 will detect regions with different inference results on N frames as anomalous regions (S504).

[0366] <<1.5. Notification Control>>

[0367] Next, the notification control according to this embodiment will be described. The control unit 150 according to this embodiment performs various notification controls related to sensor malfunctions. Hereinafter, a specific example of notification control in the case where the information processing device 10 is installed on a moving body such as a vehicle will be described.

[0368] Figure 44 This is a flowchart illustrating an example of a notification control process based on self-check results according to this embodiment.

[0369] exist Figure 44 In the example shown, firstly, the user identifies a sensor malfunction (S601).

[0370] Users who detect sensor malfunctions can activate a self-check via methods such as voice input or touch panel operation (S602).

[0371] The above self-check is a process to determine whether a sensor malfunction has occurred, and it can apply the various malfunction detection methods mentioned above to the determination.

[0372] Next, the control unit 150 performs a self-check (S603).

[0373] Note that activation is not limited to user activation, and the control unit 150 may perform self-checks periodically or based on some trigger.

[0374] If no sensor malfunction is detected during the self-check (S604: No), the control unit 150 notifies the user that no sensor malfunction was detected (S608) and terminates a series of processes.

[0375] However, if a sensor malfunction is detected during self-check (S604: Yes), the control unit 150 registers the sensing data (referred to as malfunction data) acquired by the sensor that detected the malfunction as learning data 141 (S605).

[0376] When multiple information processing devices 10 share the learning data 141, the control unit 150 can send abnormal data to a server or the like that stores and manages the learning data 141.

[0377] Next, the control unit 150 notifies the user that a sensor malfunction has been detected (S606).

[0378] In addition, the control unit 150 notifies the user of how to handle sensor malfunctions (S607).

[0379] Figure 45 This is a diagram illustrating a notification example of a response method, etc., according to this embodiment.

[0380] like Figure 45 As shown, the control unit 150 controls the output unit 130 to notify the user of the fact that an anomaly has been detected in a specific sensor, information for specifying the sensor that detected the anomaly, and response methods such as "Please perform cleaning" or "Please contact the dealer".

[0381] Next, the prompt control of the inspection method according to this embodiment will be described. Figure 46 This is a flowchart illustrating an example of the prompting control flow of the inspection method according to this embodiment.

[0382] exist Figure 46 In the example shown, firstly, the user identifies a sensor anomaly (S701).

[0383] The user who has detected the sensor malfunction will issue a sensor malfunction instruction to the system through means such as voice input or touch panel operation (S702).

[0384] Based on the instructions in step S702, the control unit 150 performs control to prompt the user with the inspection method (S703).

[0385] Figure 47 This is a diagram illustrating a prompt example of the inspection method according to this embodiment.

[0386] like Figure 47 As shown, the control unit 150 controls the output unit 130 to prompt the user with methods for checking sensor malfunctions, methods for eliminating sensor malfunctions, etc.

[0387] The user performs the inspection based on the information provided (S704).

[0388] If no sensor abnormality is found during the inspection (S705: No), the user will input the information indicating that no sensor abnormality was found into the system (S709), and the control unit 150 will end a series of processes.

[0389] However, if a sensor malfunction is confirmed during the inspection (S705: Yes), the user will instruct the system to input the information about the sensor malfunction (S706).

[0390] Based on the input in step S706, the control unit 150 registers the abnormal data as learning data 141 (S707).

[0391] In addition, the control unit 150 notifies the user of how to handle sensor malfunctions (S708).

[0392] Next, the prompt control for the reliability of the reasoning results according to this embodiment will be described. Figure 48 This is a flowchart illustrating an example of a reliability-based prompting control process according to this embodiment.

[0393] exist Figure 48 In the case of the example shown, firstly, the control unit 150 uses the inferencer 210 to perform inference processing, such as instance segmentation and semantic segmentation (S801).

[0394] Next, the control unit 150 switches to the display mode for the reliability of the inference results (S802). The switching can be performed based on the user's operation.

[0395] Next, the control unit 150 performs control to display the reliability of the inference results (S803).

[0396] Figure 49 This is a diagram illustrating a display example of the reliability of the reasoning results according to this embodiment.

[0397] exist Figure 49 In the example shown, the reliability of each instance is indicated to the user by changing the line type of the instance's outline according to its reliability. Not limited to this instance, the control unit 150 can change the color of the instance's outline based on its reliability.

[0398] The user checks the reliability displayed (S804).

[0399] If the user does not detect any sensor abnormalities (S805: No), the control unit 150 terminates a series of processes.

[0400] However, if the user discovers a sensor malfunction (S805: Yes), the user will indicate that the malfunction information has been entered into the system (S806).

[0401] Based on the input in step S806, the control unit 150 registers the abnormal data as learning data 141 (S807).

[0402] In addition, the control unit 150 notifies the user of how to handle sensor malfunctions (S808).

[0403] For example, a manual description such as "If the reliability of the instance shown in the lower right corner of the screen is consistently low, there may be an anomaly in the front right camera" can be used, allowing users to make a judgment by checking the reliability.

[0404] The notification control according to this embodiment has been described above with reference to specific examples. As mentioned above, the control unit 150 may perform a self-check in response to an instruction from the user, or may request the user to perform a check.

[0405] In addition, the control unit 150 prompts the user with the reliability of the inference results, allowing the user to intuitively grasp the stability of the sensor.

[0406] Furthermore, when anomalies are detected, registering the anomalous data as learning data can lead to further improvements in the robustness of the inference engine.

[0407] <<1.6. Variations>>

[0408] Next, variations of this embodiment will be described. Although the above mainly describes the case where the sensing data includes images, the sensing data according to this embodiment does not necessarily include images.

[0409] For example, suppose multiple IMU 318s are worn on a user's body, and the user's movement is inferred based on the sensing data acquired by the multiple IMU 318s.

[0410] In this case, by masking a portion of the sensing data acquired by multiple IMUs 318 worn on the user's body and using the sensing data as learning data 141, it is possible to learn and reproduce some of the anomalies of the IMUs 318.

[0411] As described above, the number of sensor types according to this embodiment is not necessarily multiple.

[0412] Furthermore, the technical concept of this embodiment can also be applied to source location estimation of a microphone array including multiple microphones, and direction of arrival estimation of radio waves of an array antenna including multiple antenna elements.

[0413] By masking a portion of the sensing data acquired by multiple microphones or antenna units and using the sensing data as learning data 141, it is possible to learn and reproduce the anomalies of some microphones or antenna units.

[0414] The technical concept of this embodiment is widely applicable to various inference methods based on sensed data.

[0415] <2. Hardware Configuration Examples>

[0416] Next, examples of hardware configurations according to embodiments of this disclosure will be described.

[0417] First, refer to Figure 50 An example of hardware configuration for a moving body such as a vehicle equipped with an information processing device 10 is described, which performs inference using an inferencer 210 and sensor anomaly detection using an anomaly detection model 220.

[0418] When the information processing device 10 is installed on the mobile body, such as Figure 50 As shown, the information processing device 10 may include a sensor group 310, an input device 320, an output device 330, a storage device 340, an electronic control unit (ECU) 350, a navigation device 360, and a communication module 370.

[0419] (Sensor group 310)

[0420] The sensor group 310 may include an RGB camera 312, a radar 314, a LiDAR 316, and an IMU 318.

[0421] In addition, sensor group 310 may include FMCW LiDAR 317 in addition to or in place of LiDAR 316.

[0422] (Input device 320)

[0423] The input device 320 includes a touch panel, a microphone for voice input, etc.

[0424] (Output device 330)

[0425] The output device 330 includes a display, a speaker, etc.

[0426] (Storage device 340)

[0427] Storage device 340 includes hard disk drives (HDDs), solid-state drives (SSDs), etc.

[0428] (ECU 350)

[0429] ECU 350 operates as control unit 150. ECU 350 includes a central processing unit (CPU) 352, a graphics processing unit (GPU) 354, and a memory 356. Memory 356 includes random access memory (RAM) and read-only memory (ROM).

[0430] (Navigation 360)

[0431] Navigation 360 includes a GNSS receiver 365 that receives radio waves from GNSS.

[0432] (Communication Module 370)

[0433] The communication module 370 includes a network interface and communicates with computing devices in the cloud via the network 50.

[0434] Next, we will refer to Figure 51 An example of the hardware configuration of the information processing device 10, which describes the generation of learning data 141 and the learning using learning data 141.

[0435] like Figure 51 As shown, the information processing device 10, which performs tasks such as generating learning data 141 and learning using learning data 141, includes, for example, a processor 871, a ROM 872, a RAM 873, a host bus 874, a bridge 875, an external bus 876, an interface 877, an input device 878, an output device 879, a memory 880, a driver 881, a connection port 882, and a communication device 883.

[0436] Note that the hardware configuration shown here is merely an example, and some components may be omitted. Furthermore, the information processing apparatus 10 may also include components other than those shown here.

[0437] (Processor 871)

[0438] For example, processor 871 functions as an arithmetic processing device or control device, and controls the overall operation of each component or part thereof based on various programs recorded in ROM 872, RAM 873, memory 880, or removable storage medium 901. Processor 871 includes CPU and GPU.

[0439] (ROM 872 and RAM 873)

[0440] ROM 872 is a device used to store programs to be read from processor 871, data to be used for calculations, etc. RAM 873 temporarily or permanently stores, for example, programs to be read from processor 871, various parameters that change appropriately when the program is executed, etc.

[0441] (Host bus 874, bridge 875, external bus 876 and interface 877)

[0442] The processor 871, ROM 872, and RAM 873 are interconnected via a host bus 874, for example, capable of high-speed data transfer. Meanwhile, the host bus 874 is connected via a bridge 875 to an external bus 876, which has a relatively low data transfer speed. Furthermore, the external bus 876 is connected to various components via an interface 877.

[0443] (Input device 878)

[0444] As an input device 878, for example, a mouse, keyboard, touch panel, button, switch, lever, etc., may be used. Alternatively, a remote controller (hereinafter referred to as a remote controller) capable of transmitting control signals using infrared or other radio waves may be used as an input device 878. Furthermore, the input device 878 may include a voice input device such as a microphone.

[0445] (Output device 879)

[0446] The output device 879 is, for example, a device capable of visually or audibly notifying a user of the information obtained, such as a display device (e.g., a liquid crystal display (LCD), an organic electroluminescent (EL) device), an audio output device (e.g., a speaker, headphones), a printer, a mobile phone, a fax machine, etc. Furthermore, the output device 879 according to this disclosure includes various vibration devices capable of outputting tactile stimuli.

[0447] (Memory 880)

[0448] The memory 880 is a device for storing various types of data. Examples of memory 880 include magnetic storage devices such as HDDs, semiconductor storage devices, optical storage devices, and magneto-optical storage devices.

[0449] (Driver 881)

[0450] The drive 881 is, for example, a device for reading information recorded in a removable storage medium 901 such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., or for writing information to the removable storage medium 901.

[0451] (Removable storage medium 901)

[0452] For example, removable storage medium 901 can be a digital multifunction optical disc (DVD) medium, Blu-ray (registered trademark) medium, HDDVD medium, various semiconductor storage media, etc. Needless to say, removable storage medium 901 can be, for example, an integrated circuit (IC) card with a contactless IC chip installed, an electronic device, etc.

[0453] (Connect to port 882)

[0454] Connection port 882 is, for example, a port for connecting external connection devices 902 such as Universal Serial Bus (USB) ports, IEEE 1394 ports, Small Computer System Interface (SCSI) ports, RS-232C ports, optical audio terminals, etc.

[0455] (External connection device 902)

[0456] External connection device 902 is, for example, a printer, portable music player, digital camera, digital camcorder, IC recorder, etc.

[0457] (Communication device 883)

[0458] Communication device 883 is a communication device for connecting to a network, and includes, for example, wired or wireless local area networks (LANs), Bluetooth (registered trademark), communication cards for wireless USB (WUSB), routers for optical communication, routers for asymmetric digital subscriber line (ADSL), modems for various communications, etc.

[0459] <3. Conclusion>

[0460] As described above, the information processing apparatus 10 according to embodiments of the present disclosure includes: an inference unit that performs inference based on input sensing data using an inferencer 210, the inferencer being generated by learning using masked sensing data as learning data; and an anomaly detection model 220 that detects anomalies related to the sensor that has acquired the sensing data. When the anomaly detection model 220 detects an anomaly in the sensor that has acquired the sensing data, the anomaly detection model 220 masks the sensing data acquired by the sensor and inputs the masked sensing data into the inferencer 210, and the inferencer 210 performs inference based on the masked sensing data input by the anomaly detection model 220. At least a portion of the aforementioned learning data is generated using generative AI.

[0461] The above configuration can improve the robustness of the inference engine.

[0462] While preferred embodiments of the present disclosure have been described in detail with reference to the accompanying drawings, the scope of the present disclosure is not limited to such examples. It will be apparent to those skilled in the art that various modifications or alterations will occur within the scope of the technical concept set forth in the claims, and it is understood that such modifications or alterations also fall within the scope of the present disclosure.

[0463] Furthermore, each step related to the processing described in this disclosure need not be processed sequentially in the order described in the flowchart or sequence diagram. For example, the steps related to the processing of various devices may be processed in a different order than that described, or they may be processed in parallel.

[0464] Furthermore, the series of processes performed by each device described in this disclosure can be implemented by a program stored in a non-transitory computer-readable storage medium. For example, when a computer executes a program, each program is read into RAM and executed by a processor such as a CPU. The aforementioned storage medium is, for example, a magnetic disk, optical disk, magneto-optical disk, flash memory, etc. Additionally, the aforementioned programs can be distributed via, for example, a network without using a storage medium.

[0465] Furthermore, the effects described in this specification are merely exemplary or illustrative, and not restrictive. That is, in combination with or in lieu of the above effects, the technology according to this disclosure can produce other effects that are obvious to those skilled in the art from the description in this specification.

[0466] Note that the following configurations also fall within the technical scope of this disclosure. (1)

[0468] An information processing apparatus, comprising: The inference unit is configured to perform inference based on input sensed data using an inferencer generated by learning sensed data using a mask; and An anomaly detection unit is configured to detect anomalies related to the sensor that has acquired sensing data, wherein... When the anomaly detection unit detects an anomaly in the sensor that has acquired sensing data, the anomaly detection unit masks the sensing data acquired by the sensor and inputs the sensing data into the inference unit. The inference unit performs inference based on the mask sensing data input by the anomaly detection unit, and At least part of the learning data is generated using generative AI. (2)

[0470] According to the information processing device of (1), wherein, Generative AI generates sensing data to be used as learning data based on the context of the input. (3)

[0472] According to the information processing device of (2), wherein, The context includes anomaly information related to the sensor. (4)

[0474] According to the information processing device of (2), wherein, Generative AI also generates sensing data to be used as learning data based on the annotations of the input. (5)

[0476] According to the information processing device of (2), wherein, Generative AI also generates images based on the input images to be used as learning data. (6)

[0478] According to the information processing device of (5), wherein, Generative AI generates images to be used as learning data based on an input image and parameters specifying the amount of movement of objects included in that image. (7)

[0480] According to the information processing device of (3), wherein, The context is selected based on the result of the reasoning performed by the inference engine. (8)

[0482] According to the information processing device of (7), wherein, The context is selected based on a difficult scenario, which is extracted through analysis of the results estimated by the inferencer. (9)

[0484] The information processing apparatus according to any one of (1) to (8), wherein, The anomaly detection unit detects anomalies related to the sensor that acquired the sensing data based on the changes in the acquired sensing data over time. (10)

[0486] According to the information processing device of (9), wherein, The anomaly detection unit detects anomalies related to sensors that have acquired sensing data input to the inference engine, based on the variability in the reliability of the inference performed by the inference engine. (11)

[0488] The information processing apparatus according to any one of (1) to (10), wherein, The sensed data includes images. (12)

[0490] According to the information processing device of (11), wherein, The anomaly detection unit detects anomalies related to the sensor that acquired the image based on the differences between frames of the image. (13)

[0492] According to the information processing device of (11), wherein, The anomaly detection unit detects anomalies related to the sensor that acquired the image based on the optical flow of the image. (14)

[0494] According to the information processing device of (11), wherein, The anomaly detection unit detects anomalies related to the sensor that acquired the image based on the spatial features of the image. (15)

[0496] According to the information processing device of (14), wherein, The anomaly detection unit detects anomalies related to the sensor that acquired the image based on the image's focus position. (16)

[0498] The information processing apparatus according to any one of (1) to (8), wherein, The anomaly detection unit detects anomalies related to the sensors that have acquired sensing data, based on contrastive learning. (17)

[0500] The information processing apparatus according to any one of (1) to (8), wherein, The anomaly detection unit detects sensor-related anomalies based on reliability comparisons associated with multiple sensors. (18)

[0502] The information processing apparatus according to any one of (1) to (8), wherein, The anomaly detection unit detects sensor-related anomalies by integrating inference results derived from multiple sensors. (19)

[0504] An information processing method

[0505] Executed by the processor Information processing methods include: Inference based on input sensed data is performed using an inferencer, which is generated by learning sensed data using a mask as learning data; and Sensor-related anomalies were detected and the sensor data was acquired. Anomaly detection includes masking the sensor data acquired by the sensor when an anomaly is detected in the sensor that has acquired the sensing data; Performing inference includes: performing inference based on sensed data from a mask, and At least part of the learning data is generated using generative AI. (20)

[0507] A non-transitory computer-readable storage medium containing a stored program that enables a computer to function as: Information processing apparatus, including: The inference unit is configured to perform inference based on input sensed data using an inferencer generated by learning sensed data using a mask; and An anomaly detection unit is configured to detect anomalies related to the sensor that has acquired sensing data, wherein... When the anomaly detection unit detects an anomaly in the sensor that has acquired sensing data, the anomaly detection unit masks the sensing data acquired by the sensor and inputs the sensing data into the inference unit. The inference unit performs inference based on the sensed data of the mask input by the anomaly detection unit, and At least part of the learning data is generated using generative AI.

[0508] Reference Symbol List

[0509] 10. Information processing device

[0510] 110 sensor unit

[0511] 150 control unit

[0512] 210 Inference Engine

[0513] 220 Anomaly Detection Model

[0514] 230 Image Generation AI

[0515] 312 RGB camera

[0516] 314 Radar

[0517] 316 LiDAR

[0518] 317 FMCW LiDAR

[0519] 318 IMU

[0520] 141 Learning Data

[0521] 142 Annotated Image Data

[0522] 143 Annotation Mask Data

[0523] 144 Context Data

[0524] 145. Select Context.

Claims

1. An information processing apparatus, comprising: The inference unit is configured to perform inference based on input sensed data by using an inferencer generated by learning sensed data using masked data as learning data. as well as An anomaly detection unit is configured to detect anomalies related to the sensor that has acquired sensing data, wherein... When the anomaly detection unit detects an anomaly in the sensor that has acquired sensing data, the anomaly detection unit masks the sensing data acquired by the sensor and inputs the sensing data into the inference unit. The inference unit performs inference based on the sensed data of the mask input by the anomaly detection unit, and At least a portion of the learning data was generated using generative AI.

2. The information processing apparatus according to claim 1, wherein, The generative AI generates sensing data to be used as learning data based on the input context.

3. The information processing apparatus according to claim 2, wherein, The context includes anomaly information related to the sensor.

4. The information processing apparatus according to claim 2, wherein, The generative AI also generates sensing data to be used as the learning data based on the input annotations.

5. The information processing apparatus according to claim 2, wherein, The generative AI also generates images to be used as learning data based on the input images.

6. The information processing apparatus according to claim 5, wherein, The generative AI generates images to be used as learning data based on the input image and parameters specifying the amount of movement of objects included in the image.

7. The information processing apparatus according to claim 3, wherein, The context is selected based on the result of the reasoning performed by the inferencer.

8. The information processing apparatus according to claim 7, wherein, The context is selected based on a difficult scenario, which is extracted through analysis of the results estimated by the inferencer.

9. The information processing apparatus according to claim 1, wherein, The anomaly detection unit detects anomalies related to the sensor that acquired the sensing data based on the changes in the acquired sensing data over time.

10. The information processing apparatus according to claim 9, wherein, The anomaly detection unit detects anomalies related to sensors that have acquired sensing data input to the inference engine, based on changes in the reliability of the inference performed by the inference engine.

11. The information processing apparatus according to claim 1, wherein, The sensed data includes images.

12. The information processing apparatus according to claim 11, wherein, The anomaly detection unit detects anomalies related to the sensor that acquired the image based on the differences between frames of the image.

13. The information processing apparatus according to claim 11, wherein, The anomaly detection unit detects anomalies related to the sensor that acquired the image based on the optical flow of the image.

14. The information processing apparatus according to claim 11, wherein, The anomaly detection unit detects anomalies related to the sensor that acquired the image based on the spatial characteristics of the image.

15. The information processing apparatus according to claim 14, wherein, The anomaly detection unit detects anomalies related to the sensor that acquired the image based on the image's focus position.

16. The information processing apparatus according to claim 1, wherein, The anomaly detection unit detects anomalies related to the sensors that have acquired sensing data, based on contrastive learning.

17. The information processing apparatus according to claim 1, wherein, The anomaly detection unit detects sensor-related anomalies based on reliability comparisons associated with multiple sensors.

18. The information processing apparatus according to claim 1, wherein, The anomaly detection unit detects sensor-related anomalies based on the integration of inference results derived from multiple sensors.

19. An information processing method Executed by the processor, The information processing method includes: Inference based on input sensed data is performed by using an inferencer, which is generated by learning sensed data using a mask as learning data; as well as Sensor-related anomalies were detected and the sensor data was acquired. Anomaly detection includes: when an anomaly is detected in a sensor that has acquired sensing data, masking the sensing data acquired by the sensor; Performing inference includes: performing inference based on sensed data from a mask, and At least a portion of the learning data was generated using generative AI.

20. A non-transitory computer-readable storage medium storing a program, the program causing a computer to function as: Information processing apparatus, including: The inference unit is configured to perform inference based on input sensed data by using an inferencer generated by learning sensed data using masked data as learning data. as well as An anomaly detection unit is configured to detect anomalies related to the sensor that has acquired sensing data, wherein... When the anomaly detection unit detects an anomaly in the sensor that has acquired sensing data, the anomaly detection unit masks the sensing data acquired by the sensor and inputs the sensing data into the inference unit. The inference unit performs inference based on the sensed data of the mask input by the anomaly detection unit, and At least a portion of the learning data was generated using generative AI.

Images