Learning model generation method, information processing device, scene determination system, learning program, and scene determination program

By integrating camera images with visualized audio data, the method addresses the challenge of diverse data types in learning models, improving efficiency and accuracy for scene recognition in vehicles.

JP2024166749AInactive Publication Date: 2024-11-29DENSO TEN LTD
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
JP2023083069
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-19
Publication Date
2024-11-29
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Existing learning models struggle to efficiently accommodate and process diverse types and sizes of data, such as audio signals and sensing data, for scene recognition in vehicles.

Method used

A method is introduced to create a learning model by integrating camera images with visualized audio data, using techniques like wavelet transform or FFT, to generate integrated data that includes correct labels, facilitating machine learning for scene recognition.

Benefits of technology

This approach allows for efficient creation and deployment of learning models that reduce processing load and time, enhancing scene recognition accuracy and suitability for real-time vehicle applications.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024166749000001_ABST
    Figure 2024166749000001_ABST
Patent Text Reader

Abstract

To provide a learning model generation method, an information processing device, a scene determination system, a learning program, and a scene determination program for obtaining efficiency in generating a learning model with the use of different kinds of data.SOLUTION: A learning model generation method for identifying a scene of image data with respect to the image data includes steps for: generating second image data obtained by visualizing sound data acquired in a sound acquisition period corresponding to a photographing time point of the first image data obtained by photographing with the use of a camera (S2); generating third image data obtained by integrating the first image data with the second image data (S3); and performing, in the third image data, learning to generate a learning model by learning data to which the scene corresponding to the first image data is imparted as correct answer data (S4).SELECTED DRAWING: Figure 8
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a learning model generation method for generating a learning model for scene recognition, an information processing device for performing scene recognition, a scene determination system, a learning program, and a scene determination program. [Background technology]

[0002] Conventionally, a driving assistance system has been proposed that includes a plurality of vehicles equipped with a plurality of microphones and sensors, and a server having an acquisition unit that acquires audio signals recorded by the plurality of microphones and sensing data measured by the sensors (for example, Patent Document 1). The server includes a storage unit that stores learning data in which audio signals and sensing data are associated with information indicating the danger of a sound source, a model generation unit that uses the learning data to generate a learning model that predicts the danger of a sound source based on the audio signals and sensing data, and a provision unit that provides the danger to the plurality of vehicles. It is also disclosed that the server further includes a photographing unit that photographs an image of the sound source, and accumulates the image as learning data in association with information indicating the danger of the sound source. That is, the sound source is recognized using an image, and information indicating the danger of the sound source is generated. The information indicating the danger is used for annotation of the learning data. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-144404 A Summary of the Invention [Problem to be solved by the invention]

[0004] When generating learning data and a learning model using multiple data such as voice signals and sensing data acquired from a vehicle, it is necessary to configure the input part of the learning model so as to accommodate various data types, sizes, etc. Therefore, the present disclosure aims to improve the efficiency of creating a learning model using different types of data. [Means for solving the problem]

[0005] A learning model generation method according to the present disclosure is a method for generating a learning model for identifying a scene of image data from image data. The learning model generation method creates second image data that visualizes voice data acquired during a voice acquisition period corresponding to a time point when first image data captured by a camera was captured, creates third image data by integrating the first image data and the second image data, and performs learning to generate the learning model using learning data in which a scene corresponding to the first image data is added as correct answer data to the third image data. Effect of the Invention

[0006] According to the present technology, it is possible to efficiently create learning models using different types of data. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a block diagram illustrating an example of a system configuration. [Diagram 2] FIG. 2 is a diagram showing an example of image data captured by a camera. [Diagram 3] FIG. 3 shows an example of a visualized image created by applying wavelet transform to a sound. [Figure 4] FIG. 4 is a diagram showing an example of a visualized image created by FFT. [Diagram 5] FIG. 5 is a diagram illustrating an example of the integrated data. [Figure 6] FIG. 6 is a diagram showing integrated data according to a modified example. [Figure 7] FIG. 7 is a diagram showing integrated data according to a modified example. [Figure 8] FIG. 8 is a process flow diagram illustrating an example of the learning process. [Figure 9] FIG. 9 is a process flow diagram showing an example of the scene recognition process. [Figure 10]FIG. 10 is a diagram for explaining the result of scene recognition that is output. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] <Embodiment> Hereinafter, an embodiment will be described with reference to the drawings. FIG. 1 is a block diagram for explaining an example of the configuration of a system. The system 100 includes a vehicle 1 and an information processing device 2. The vehicle 1 performs scene recognition using image data captured by a camera, audio data acquired by a microphone, and a trained model created by the information processing device 2 through machine learning. The trained model is a classifier for detecting scenes such as dangerous situations, such as the approach of a car or a pedestrian at an intersection. The trained model is created by machine learning using input (question) data consisting of an image in which a predetermined situation occurs during driving and audio for a predetermined period corresponding to the image capture time point, and a label (correct answer data) representing a scene as teacher data.

[0009] The vehicle 1 is a passenger car or the like. The vehicle 1 includes a processor 11, a storage device 12, a camera 13, a microphone 14, and a user interface (UI) 15. These components are connected via a bus to configure a scene determination system. The vehicle 1 performs scene recognition according to this embodiment while traveling.

[0010] The processor 11 is an arithmetic processing device such as a CPU (Central Processing Unit). The processor 11 executes a program to perform each process according to the embodiment. In the example of Fig. 1, functional blocks are shown in the processor 11. That is, the processor 11 executes a predetermined program to function as a preprocessing unit 111 and a determination unit 112. These functional units will be described later.

[0011] The storage device 12 is a main storage device such as a random access memory (RAM) or a read only memory (ROM), and an auxiliary storage device such as a hard disk drive (HDD), a solid state drive (SSD), or a flash memory. The storage device 12 temporarily stores programs read by the processor 11 and information to be processed, and secures a working area for the processor 11. The storage device 12 is also assumed to store a trained model 221. The storage device 12 is also assumed to store image data captured by the camera 13, audio data acquired by the microphone 14, or data obtained by converting these data through a predetermined pre-processing.

[0012] The camera 13 is an imaging device that converts light into an electric signal using an image sensor using, for example, a CCD or a CMOS, creates image data, and outputs the image data. FIG. 2 is a diagram showing an example of image data captured by the camera. The camera 13 creates a camera image 3 that captures, for example, the traveling direction of the vehicle 1. The camera image 3 corresponds to one still image that constitutes a video. That is, the camera image 3 is continuously created. The camera 13 may be a so-called wide-angle lens, and the angle of view of the camera image 3 is not limited to the example of FIG. 2. The camera 13 may also be a so-called omnidirectional camera. For example, the camera 13 may include two imaging devices mounted on the vehicle 1 facing the traveling direction of the vehicle 1 and the opposite direction, and may output both the camera images in the traveling direction and the opposite direction, or may output a stitched camera image 3.

[0013] The microphone 14 is an input device that converts voice into an electric signal and outputs it by using a method such as a dynamic type, a capacitor type, or a piezoelectric type. The microphones 14 are mounted on the vehicle 1 so as to capture voice. The number of microphones 14 is not particularly limited. The characteristics of the microphones 14, such as their directivity, are also not particularly limited. However, when a plurality of microphones 14 are provided, it is preferable to arrange the microphones 14 with relatively high directivity so as to face in different directions.

[0014] The UI 15 may be a display or a speaker, etc. The UI 15 outputs the results of the scene recognition.

[0015] The information processing device 2 is a computer such as a workstation, a server, or a PC (Personal Computer). The information processing device 2 is a computer. The information processing device 2 includes a processor 21 and a storage device 22. These components are connected via a bus. The processor 21 is also an arithmetic processing device such as a CPU. The information processing device 2 performs so-called machine learning to create a trained model for performing scene recognition according to this embodiment.

[0016] The processor 21 executes a program to perform each process according to the embodiment. In the example of Fig. 1, functional blocks are shown in the processor 21. That is, the processor 21 executes a predetermined program to function as a preprocessing unit 211 and a learning unit 212. These functional units will be described later.

[0017] The storage device 22 is a main storage device such as a RAM or a ROM, and an auxiliary storage device such as a HDD, an SSD, or a flash memory. The main storage device temporarily stores the programs read by the processor 21 and information to be processed, and secures a working area for the processor 21. The auxiliary storage device stores the programs executed by the processor 21, information acquired from other computers, and the like. For example, the storage device 22 stores image data output by the camera 13 equipped in the vehicle 1 and audio data output by the microphone 14 as learning data. In addition, it is preferable that the learning data is collected from a plurality of vehicles 1.

[0018] The pre-processing unit 111 of the vehicle 1 and the pre-processing unit 211 of the information processing device 2 perform pre-processing on the data to be processed. The pre-processing unit 111 and the pre-processing unit 211 integrate the image data and the voice data into one image data. In other words, the voice data is visualized. The pre-processing unit 111 and the pre-processing unit 211 perform frequency analysis such as wavelet transformation and FFT (Fast Fourier Transformation) on the voice data for a predetermined period of time, and display the results. An image is generated. The predetermined period may be a period that allows the situation around the vehicle to be determined, such as 5 seconds. The predetermined period may also be a period that starts a predetermined time before the capture of the corresponding image data and ends at the capture of the image data. FIG. 3 shows an example of a visualized image 4 created by wavelet transforming the sound. The vertical axis of FIG. 3 represents frequency. The horizontal axis of FIG. 3 represents time. The color scale of FIG. 3 represents the intensity of the sound. Note that the parameters corresponding to each axis are not limited to this example. FIG. 4 is a diagram showing an example of a visualized image 4A created by FFT. The vertical axis of FIG. 4 represents the intensity of the sound. The horizontal axis of FIG. 4 represents the frequency. Note that the parameters of the vertical and horizontal axes may be reversed.

[0019] Then, the preprocessing unit 111 and the preprocessing unit 211 integrate the image data at the time corresponding to the audio data. The preprocessing unit 111 and the preprocessing unit 211 perform the same type of image data integration process. FIG. 5 is a diagram showing an example of integrated data 5. In the example of FIG. 5, the camera image 3 and the visualized image 4 are superimposed to form one image. The integrated data 5 can be created by superimposing one of the camera image 3 and the visualized image 4 on the other image whose alpha channel has been changed to a predetermined transparency (transparency). The integrated data 5 may be a camera image 3 and a visualized image 4 arranged side by side at different positions (i.e., not overlapping). FIG. 6 is a diagram showing integrated data 5A relating to the latter integrated data creation method. In the example of FIG. 6, the camera image 3 and the visualized image 4 are arranged adjacent to each other vertically to form one integrated data. A visualized image 5A is formed. In the examples of Figs. 5 and 6, the visualized image 4A and the camera image 3 may be integrated instead of the visualized image 4. When using voice data output from a plurality of microphones 14, the visualized images 4 corresponding to the plurality of microphones 14 are arranged at different positions so as not to overlap with each other, and integrated data is created. Fig. 7 is a diagram showing integrated data 5B related to this integrated data creation method. In the example of Fig. 7, for example, for a camera image capturing the forward direction of the vehicle 1, a visualized image 4B based on voice data acquired by a microphone 14 provided on the left side facing the forward direction (a microphone 14 having directionality toward the forward left side of the vehicle 1) is superimposed on the left side of the camera image 3. In addition, a visualized image 4C based on voice data acquired by a microphone 14 provided on the right side facing the forward direction of the vehicle 1 (a microphone 14 having directionality toward the forward right side of the vehicle 1) is superimposed on the right side of the camera image 3. The visualized image may also be superimposed on an image in the rear direction of the vehicle 1 in the same manner as in the forward direction to create integrated data 5. In this case, visualized images created based on audio data acquired from a microphone 14 with directionality in the same direction as the left-right direction of the camera image are superimposed on the left and right of the camera image. By aligning the sound source directions of the image and audio in this way, the image of the integrated data 5 will contain data components of the sound source direction, and the characteristics of the sound source direction can also be learned, improving the accuracy of the determination.

[0020] The learning unit 212 of the information processing device 2 performs machine learning using integrated learning data based on the integrated data 5 after preprocessing, and creates a learned model 221. The integrated learning data used as learning data (training data) is the integrated data 5 to which a correct label representing a scene has been added. For example, the camera image 3 in FIG. 2 is annotated in advance based on, for example, a user's operation, and a correct label is added. In this case, the correct label may be carried over during the integration process and added to the integrated data 5. In addition, a correct label that is not shown in the camera image 3 and is recognized from the visualized image 4 (i.e., audio data) may also be associated with, for example, the camera image 3 or the visualized image 4 (for example, annotation is performed in advance based on a user's operation). In this case, too, the correct label may be carried over during the integration process and added to the integrated data 5. By using the integrated data 5 obtained by integrating such a camera image 3 and a visualized image 4, the relationship between the characteristics of the captured image data and the characteristics of the audio at the time of capture can be learned.

[0021] The correct label may be assigned to a bounding box that indicates an area where an object exists in the camera image 3 or the integrated data 5. In this case, for example, an object detection area (area where an object exists) based on a camera image, the camera image, and a visualized image of the sound are used as input data, and machine learning is performed using learning data with the correct label as correct data.

[0022] The correct label depends on the usage form of the trained model 221, but may represent a scene such as the presence or absence of an approaching obstacle that impedes vehicle travel, the degree of the approach, the presence or absence of a risk of an accident, or the degree of the risk, or may be data used for processing of scene determination. The trained model 221 created by the learning unit 212 is mounted on the vehicle 1 in the operation stage and used for vehicle travel assistance, etc.

[0023] The determination unit 112 of the vehicle 1 performs scene recognition using the integrated data 5 after preprocessing. In the operation stage, the integrated data 5 to be recognized is not associated with a correct label. The determination unit 112 performs scene recognition for the integrated data 5 using the trained model 221. For example, the determination unit 112 determines the scene for the camera image 3 at the time of determination, such as the presence or absence of a risk of an accident occurring or the degree of the risk, from the integrated data 5. In addition, the determination unit 112 outputs the result of the scene recognition via the UI 15 of the vehicle 1.

[0024] <Learning process> FIG. 8 is a process flow diagram showing an example of the learning process. It is assumed that the camera images 3 and audio data collected by a plurality of vehicles 1 are stored in the storage device 22 of the information processing device 2 in advance. It is also possible to use a data collection method in which a communication device is installed in the vehicle 1 equipped with a scene determination system, and the image and audio data acquired by the vehicle 1 (scene determination system) with the camera 13 and microphone 14 is transmitted to the information processing device 2 by communication via the communication device. It is also assumed that the camera images 3 are previously annotated by a user or the like, and a correct answer label representing the captured scene is associated with the camera images 3. It is assumed that the learning process is started by the person in charge of creating the trained model 221 performing a learning start operation or the like (detection and trigger of the operation).

[0025] First, the pre-processing unit 211 (processor) of the information processing device 2 acquires the camera image 3 and the voice data from the storage device 22 (FIG. 8: step S1). The pre-processing unit 211 also performs, for example, frequency analysis on the voice data acquired in step S1 to create a visualized image 4 that represents the characteristics of the voice data (FIG. 8: step S2). The frequency analysis may be, for example, wavelet transformation or FFT. The pre-processing unit 211 also integrates the camera image 3 acquired in step S1 and the visualized image 4 created in step S2 to create integrated data 5 (FIG. 8: step S3). The visualized image 4 is given the correct answer level given to the camera image 3 acquired in step S1. In this step S3, integrated learning data is created in which the correct answer level is given to the integrated data 5 as shown in, for example, FIG. 5 to FIG. 7. Although not shown, the processes of steps S1 to S3 are sequentially performed on each camera image 3 and voice data for creating learning data stored in the storage device 22, and a learning data set of multiple integrated learning data is created.

[0026] Then, the learning unit 212 (processor) of the information processing device 2 performs machine learning of the scene using the created learning data set to create a learned model (FIG. 8: step S4). Machine learning can be performed by existing methods using neural networks, such as so-called deep learning. In particular, it may include methods suitable for image recognition, such as convolutional neural networks. In addition, by using integrated learning data, scene recognition based on the characteristics of not only the camera image 3 but also the audio data can be learned. Note that, according to this embodiment, learning is performed based on image data (including visualized images of audio). Since learning based on image data has been relatively well researched and developed, high-quality learning can be expected according to the learning according to this embodiment. In addition, since the image data of the captured image and the visualized image of the audio are synthesized, learning processing is performed only on image data, and therefore, compared to learning based on learning data including both image data and audio data, it is expected that the processing load will be lighter.

[0027] Then, the learning unit 212 evaluates the performance of the trained model (FIG. 8: step S5). In this step S5, the learning unit 212 performs scene recognition on, for example, test data other than the integrated training data (for example, the integrated training data is divided into data for training and data for evaluation, and the data for evaluation is used) using the model created in step S4, and verifies the accuracy of the recognition. Note that a correct answer label is also associated in advance with the test data for comparison with the scene recognition result. The verification may be performed by calculating an existing evaluation index such as a correct answer rate. Note that, for example, the learning data, etc. may be changed and the learning process may be repeated until a desired standard (correct answer rate threshold) is satisfied.

[0028] <Scene recognition processing> 9 is a process flow diagram showing an example of a scene recognition process executed by the vehicle 1 in the operation stage. The process in FIG. 9 is started, for example, when the ACC of the vehicle 1 is turned on (or triggered by engine start), and is repeatedly executed while the vehicle 1 is traveling. The camera 3 continuously outputs image data (camera image 3) and stores it in the storage device 12. Similarly, the microphone 14 continuously outputs voice data and stores it in the storage device 12. Also, it is assumed that the storage device 12 stores a trained model 221 created in advance by the information processing device 2.

[0029] First, the pre-processing unit 111 (processor) of the vehicle 1 acquires the camera image 3 and audio data from the storage device 12 (FIG. 9: step S11). When the scene recognition result is used for vehicle driving support, real-time performance is required, so the latest image data captured by the camera 13 is acquired from the storage device 12 (or acquired directly from the camera 13 (sometimes via a buffer memory or the like)). As for the audio data, data for a predetermined time (e.g., 5 seconds) immediately before the capture time of the corresponding image data is acquired from the storage device 12.

[0030] The pre-processing unit 111 also performs, for example, frequency analysis to create a visualized image 4 that represents the characteristics of the audio data (FIG. 9: step S12). The frequency analysis is performed by the same method as S2 in FIG. 8. For example, wavelet transform, FFT, etc. are performed (parameters used in the processing are also the same). The pre-processing unit 111 also integrates the camera image 3 and the visualized image 4 to create integrated data 5 (FIG. 9: step S13). The processing in this step S13 is the same as S3 in FIG. 8, but no processing for assigning a correct answer level is performed. Then, the determination unit 112 (processor) of the vehicle 1 applies (inputs) the integrated data 5 to the trained model 221 to perform scene recognition (FIG. 9: step S14). In this step S14, the integrated data 5 is input to the trained model 221, and information indicating, for example, the presence or absence of a risk of an accident occurring or the degree of the risk is obtained as an output. The determination unit 112 also performs processing according to the result of the scene recognition (FIG. 9: step S15). The determination unit 112 outputs, for example, an identification result. The output is performed via the UI 15 such as a display or a speaker. The output may be performed, for example, when it is determined that there is a risk of an accident occurring, or when it is determined that the degree of risk exceeds a predetermined threshold. In addition, in the learning stage, when the correct answer label associated with the bounding box is used as the teacher value, data such as the position of the recognized (detected) object and the object type (recognition result data based on the area of ​​the bounding box and the information on the correct answer level) are output as the recognition result. Therefore, in this case, it is possible to perform obstacle notification and vehicle driving assistance according to the position of the obstacle. In addition, the determination unit 112 may perform control in so-called automatic driving using the identification result. In addition, the scene recognition result output by the determination unit 112 may be input to a separately provided vehicle control device to perform various vehicle driving controls, vehicle driving assistance controls, and the like.

[0031] FIG. 10 is a diagram for explaining the result of the scene recognition to be output. In the example of FIG. 10, the result of the scene recognition using the integrated data 5 is superimposed on the camera image 3 and displayed on the UI 15. In the result image 6A, an arrow 61 indicating a direction to be careful in, obtained as a result of the scene recognition, is superimposed on the camera image 3 at a certain time. The determination unit 112 may output information indicating a scene not shown in the camera image 3 but recognized based on the visualized image 4 of the sound around the vehicle, for example, by a figure or a symbol indicating a direction. Here, the arrow 61 indicates the presence of a vehicle (presumed by sound) approaching the vehicle from the left rear. It is also possible to detect the approach of a pedestrian not shown in the visualized image 4 but captured in the camera image 3. In the result image 6B, bounding boxes 62 and 63 indicating the presence and position of a potentially dangerous subject obtained as a result of the scene recognition are superimposed on the camera image 3 at a later time. Here, a passenger car that has overtaken the vehicle from the left rear is indicated by the bounding box 62. Also, a truck approaching the vehicle from the right front is indicated by a bounding box 63. For example, such a scene is recognized using the trained model 221 and the integrated data 5, and the result is displayed on the display of the UI 15. Note that the output of the result is not visual information but is displayed through a speaker. The notification may be made by voice via a US15 such as the one shown in FIG. 1, and may be used for vehicle driving control or vehicle driving assistance.

[0032] <Effects> According to this embodiment, machine learning is performed based on integrated data in which the camera image 3 and the voice data are integrated into image data, and scene recognition is performed using the integrated data. Therefore, the load of the processing related to model creation and scene recognition and the time required for processing can be reduced compared to, for example, creating a learning model for images and a learning model for voice separately and determining a scene using both of these models. In other words, it is possible to efficiently create a learning model using different types of data. In addition, the learned model is installed in the vehicle 1, and can be determined without, for example, communication. Therefore, it can be suitably used for applications in which a quick determination is required depending on the situation.

[0033] Furthermore, if the integrated data 5 is created by superimposing the camera image 3 and the visualized image 4 as shown in Fig. 5, the image size can be reduced, and the processing load can be reduced. On the other hand, if the integrated data 5 is created by arranging the camera image 3 and the visualized image 4 in different positions as shown in Fig. 6, the features of the camera image 3 and the visualized image 4 can be used for learning and judgment without being mixed up or diluted.

[0034] In addition, instead of superimposing the camera image 3 and the visualized image 4 by changing the transparency, a superimposed image may be created by assigning different color components, such as RGB, to each of the camera image 3 and the visualized image 4.

[0035] <Modification> When using the integrated data 5 in which the camera image 3 and the visualized image 4 are superimposed, the ratio of the camera image 3 and the visualized image 4 to be superimposed may be changed depending on, for example, the surrounding environment of the point where the camera image 3 is captured. The surrounding environment may be, for example, the brightness of the camera image 3 or the type of the location where the image is captured. The brightness of the camera image 3 can be determined based on, for example, the date and time when the camera image 3 was created, or color information of an area recognized as the sky in the camera image 3. The type of the location where the image is captured can be determined based on, for example, an imaged object recognized in the camera image 3 or meta information indicating the location where the camera image 3 was created (position information obtained from a satellite positioning system or a car navigation system connected thereto is added as meta information to the camera image data). In addition, when it is determined that the sound data obtained from the microphone 14 contains a noise component equal to or greater than a predetermined threshold, the ratio of the visualized image 4 may be reduced.

[0036] When implementing the above-mentioned modified example, the learning process and the scene recognition process are performed as follows. In step S3 of FIG. 8, the transparency of one of the camera image 3 and the visualized image 4 to be superimposed on the other image is changed in various ways to create a plurality of integrated data 5, and integrated learning data is created based on the integrated data 5. In step S4 of FIG. 8, learning is performed using the integrated learning data generated under each transparency condition for each surrounding environment state. Then, in step S5 of FIG. 8, for each learning model generated in each of the above learning processes, a learning model trained under the transparency condition that is evaluated most highly for each surrounding environment is selected, and a data table is generated in which the selected learning model, the corresponding transparency condition, and the corresponding surrounding environment are associated. Then, this data table and each selected learning model are recorded in the storage device 22 of the information processing device 2. Meanwhile, in step S13 of FIG. 9, the surrounding environment of the point where the camera image 3 to be judged was captured is detected, and the transparency condition and learning model corresponding to the detected surrounding environment are extracted based on the data table, and the learning model to be used for scene recognition and the transparency information to be used for image integration processing are determined. Then, based on the determined transparency, the ratio of superimposition of the camera image 3 and the visualized image 4 is changed. The integrated image is applied to the selected learning model to identify the scene.

[0037] <Other> The illustrated system and device configurations are merely examples, and are not limited to the above examples. For example, at least some of the functional units of the information processing device 2 may be shared and executed by multiple devices, or may be executed in parallel by multiple devices. Similarly, the processes shown in the processing flow may be executed in a different order or in parallel, as long as the results are the same.

[0038] In addition, it is preferable that the visualized image 4 in the learning stage and the visualized image 4 in the operation stage are drawn so that the scale of each axis is the same. However, for example, an anchor box including the feature of the visualized image 4 in the learning stage may be set so that scene recognition can be performed from visualized images 4 with different aspect ratios or sizes.

[0039] The present invention also includes a computer program for executing the above-mentioned processing method, and a computer-readable recording medium on which the program is recorded. The above-mentioned processing is possible by having a computer read the recording medium and execute the recorded program. A computer-readable recording medium is a recording medium that stores information such as data and programs by electrical, magnetic, optical, mechanical, or chemical action and can be read by a computer. Among such recording media, those that can be removed from a computer include flexible disks, magneto-optical disks, optical disks, magnetic tapes, memory cards, etc. Furthermore, recording media fixed to a computer include hard disk drives and ROMs, etc.

[0040] The gist of the present disclosure is as follows. (Aspect 1) A method for generating a learning model for identifying a scene of image data, the method comprising the steps of: creating second image data that visualizes voice data acquired during a voice acquisition period corresponding to a capture time point of the first image data captured by the camera; creating third image data by integrating the first image data and the second image data; performing learning for generating the learning model using learning data in which a scene corresponding to the first image data is added as correct answer data to the third image data; Learning model generation method. (Aspect 2) The third image data is created by superimposing the first image data and the second image data. A learning model generation method according to aspect 1. (Aspect 3) The ratio of superimposition of the first image data and the second image data is changed according to the surrounding environment of the point where the first image data is captured. A learning model generation method according to aspect 2. (Aspect 4) the audio data includes a plurality of pieces of audio data acquired via a plurality of microphones acquiring audio from different directions; creating a plurality of second image data by visualizing the plurality of voice data, Each of the plurality of second image data is created by arranging the second image data at a different position relative to the first image data. A learning model generation method according to aspect 1. (Aspect 5) The audio acquisition period is a period that starts a set time before the capture of the first image data and ends at the capture of the first image data. A learning model generation method according to aspect 1. (Aspect 6) The visualization is performed by generating an image representing the results of the frequency analysis. A learning model generation method according to aspect 1. (Aspect 7) An information processing device for identifying a scene of image data, the information processing device comprising: The learning model includes an input data set that is an integrated image data set for learning obtained by integrating learning image data captured by a camera with audio image data that visualizes learning audio data acquired during an audio acquisition period corresponding to the time point when the learning image data was captured, and a trained model trained using the learning data that is an input data set that is assigned a scene corresponding to the learning image data as correct answer data; Acquire image data for identification taken by the camera, Acquire voice data for identification during a voice acquisition period corresponding to the time point when the identification image data was captured; creating integrated image data for identification by integrating the image data for identification with the voice image data obtained by visualizing the voice data for identification; Applying the integrated image data for discrimination to the trained model to discriminate the scene. Information processing device. (Aspect 8) A microphone for capturing surrounding sounds; A camera that captures the surroundings, A scene determination system to be mounted on a vehicle, the system including an information processing device that receives audio data output by the microphone and image data output by the camera and identifies a scene of the image data, The information processing device includes: The learning model includes an input data set that is an integrated image data set for learning obtained by integrating learning image data captured by a camera with audio image data that visualizes learning audio data acquired during an audio acquisition period corresponding to the time point when the learning image data was captured, and a trained model trained using the learning data that is an input data set that is assigned a scene corresponding to the learning image data as correct answer data; Acquire image data for identification taken by the camera, Acquire voice data for identification during a voice acquisition period corresponding to the time point when the identification image data was captured; creating integrated image data for identification by integrating the image data for identification with the voice image data obtained by visualizing the voice data for identification; Applying the integrated image data for discrimination to the trained model to discriminate the scene. Scene determination system. (Aspect 9) A learning program executed by a computer to generate a learning model for identifying a scene in image data, comprising: creating second image data that visualizes voice data acquired during a voice acquisition period corresponding to a capture time point of the first image data captured by the camera; creating third image data by integrating the first image data and the second image data; A learning model is trained using training data in which a scene corresponding to the first image data is added as correct answer data to the third image data, and the learning model is generated. Study program. (Aspect 10) A scene determination program executed by a computer to identify a scene in image data, comprising: The learning data is an integrated image data for learning obtained by integrating learning image data for learning captured by a camera with audio image data obtained by visualizing learning audio data for learning acquired during an audio acquisition period corresponding to the time point at which the learning image data was captured, and a trained model is trained using the training data in which a scene corresponding to the learning image data is added as correct answer data. Acquire image data for identification taken by the camera, Acquire voice data for identification during a voice acquisition period corresponding to the time point when the identification image data was captured; creating integrated image data for identification by integrating the image data for identification with the voice image data obtained by visualizing the voice data for identification; Applying the integrated image data for discrimination to the trained model to discriminate the scene. Scene judgment program. [Explanation of symbols]

[0041] 100: System 1: vehicle, 11: processor, 111: pre-processing unit, 112: determination unit, 12: storage device, 13: camera, 14: microphone, 15: user interface 2: information processing device, 21: processor, 211: preprocessing unit, 212: learning unit, 22: storage device, 221: trained model 3: Camera image, 4(4A, 4B, 4C): Visualized image, 5(5A, 5B): Integrated data, 6(6A, 6B): Result image

Claims

1. A method for generating a learning model for identifying a scene of image data, the method comprising the steps of: creating second image data that visualizes voice data acquired during a voice acquisition period corresponding to a capture time point of the first image data captured by the camera; creating third image data by integrating the first image data and the second image data; performing learning for generating the learning model using learning data in which a scene corresponding to the first image data is added as correct answer data to the third image data; Learning model generation method.

2. The third image data is created by superimposing the first image data and the second image data. The learning model generating method according to claim 1 .

3. A ratio of superimposition of the first image data and the second image data is changed according to the surrounding environment of the point where the first image data is captured. The learning model generating method according to claim 2 .

4. the audio data includes a plurality of pieces of audio data acquired via a plurality of microphones acquiring audio from different directions; creating a plurality of second image data by visualizing the plurality of voice data, The third image data is generated by arranging each of the plurality of second image data at a different position relative to the first image data. The learning model generating method according to claim 1 .

5. The audio acquisition period is a period that starts a set time before the capture of the first image data and ends at the capture of the first image data. The learning model generating method according to claim 1 .

6. The visualization is performed by generating an image representing the results of the frequency analysis. The learning model generating method according to claim 1 .

7. An information processing device for identifying a scene of image data, the information processing device comprising: The learning model is provided with an input data set of integrated learning image data obtained by integrating learning image data captured by a camera with learning audio image data obtained by visualizing learning audio data acquired during an audio acquisition period corresponding to the time point of capturing the learning image data, and a trained model trained with learning data obtained by adding a scene corresponding to the learning image data as correct answer data to the input data, Acquire image data for identification taken by the camera, Acquire voice data for identification during a voice acquisition period corresponding to the time point when the identification image data was captured; creating integrated image data for identification by integrating the image data for identification with the voice image data obtained by visualizing the voice data for identification; Applying the integrated image data for discrimination to the trained model to discriminate the scene. Information processing device.

8. A microphone for capturing surrounding sounds; A camera that captures the surroundings, A scene determination system to be mounted on a vehicle, the system including an information processing device that receives audio data output by the microphone and image data output by the camera and identifies a scene of the image data, The information processing device includes: The learning model is provided with an input data set of integrated learning image data obtained by integrating learning image data captured by a camera with learning audio image data obtained by visualizing learning audio data acquired during an audio acquisition period corresponding to the time point of capturing the learning image data, and a trained model trained with learning data obtained by adding a scene corresponding to the learning image data as correct answer data to the input data, Acquire image data for identification taken by the camera, Acquire voice data for identification during a voice acquisition period corresponding to the time point when the identification image data was captured; creating integrated image data for identification by integrating the image data for identification with the voice image data obtained by visualizing the voice data for identification; Applying the integrated image data for discrimination to the trained model to discriminate the scene. Scene determination system.

9. A learning program executed by a computer to generate a learning model for identifying a scene in image data, comprising: creating second image data that visualizes voice data acquired during a voice acquisition period corresponding to a capture time point of the first image data captured by the camera; creating third image data by integrating the first image data and the second image data; A learning model is trained using training data in which a scene corresponding to the first image data is added as correct answer data to the third image data, and the learning model is generated. Study program.

10. A scene determination program executed by a computer to identify a scene in image data, comprising: The learning data is an integrated image data for learning obtained by integrating learning image data for learning captured by a camera with audio image data obtained by visualizing learning audio data for learning acquired during an audio acquisition period corresponding to the time point at which the learning image data was captured, and a trained model is trained using the training data in which a scene corresponding to the learning image data is added as correct answer data. Acquire image data for identification taken by the camera, Acquire voice data for identification during a voice acquisition period corresponding to the time point when the identification image data was captured; creating integrated image data for identification by integrating the image data for identification with the voice image data obtained by visualizing the voice data for identification; Applying the integrated image data for discrimination to the trained model to discriminate the scene. Scene determination program.

Citation Information

Patent Citations

  • Driving support system

    JP2020144404A