Model generation device, conversion device, model generation method, conversion method, and program

Abstract

This model generation device comprises a training unit. The training unit generates a model by performing machine learning by using a first image and a second image. The first image is an image generated by using first radar signal data. The second image is an image generated by using second radar signal data. The first radar signal data is a signal extracted from the second radar signal data.

Description

Model generation device, conversion device, model generation method, conversion method, and recording medium

[0001] The present invention relates to a model generation device, a conversion device, a model generation method, and a program.

[0002] Radar can transmit and receive radio waves to and from a target, thereby creating an image of the target.

[0003] Patent Document 1 describes a technology that uses radar to determine the possibility that a subject is in possession of a dangerous substance. Patent Document 1 also describes performing secondary screening using radar, which allows for more accurate detection than primary screening.

[0004] Japanese Patent Application Laid-Open No. 2022-164899

[0005] However, in the above-mentioned Patent Document 1, the resolution of the image obtained by the radar depends on the number of antennas and the frequency of the radar.

[0006] In view of the above-mentioned problems, one example of the object of the present invention is to provide a model generation device, a conversion device, a model generation method, and a program that make it possible to obtain high-resolution images regardless of the radar antenna configuration.

[0007] According to one aspect of the present invention, there is provided a model generation device including: a learning means for generating a model by performing machine learning using a first image and a second image, wherein the first image is an image generated using first radar signal data; the second image is an image generated using second radar signal data; and the first radar signal data is a signal extracted from the second radar signal data.

[0008] According to one aspect of the present invention, there is provided a conversion device including conversion means for converting a target image using a model generated by the above-described model generation device.

[0009] According to one aspect of the present invention, there is provided a model generation method, in which one or more computers generate a model by performing machine learning using a first image and a second image, wherein the first image is an image generated using first radar signal data, the second image is an image generated using second radar signal data, and the first radar signal data is a signal extracted from the second radar signal data.

[0010] According to one aspect of the present invention, there is provided a transformation method in which one or more computers transform a target image using a model generated by the model generation device described above.

[0011] According to one aspect of the present invention, there is provided a computer-readable recording medium having a program recorded thereon, the program causing a computer to function as a learning means for generating a model by performing machine learning using a first image and a second image, the first image being an image generated using first radar signal data, the second image being an image generated using second radar signal data, and the first radar signal data being a signal extracted from the second radar signal data.

[0012] According to one aspect of the present invention, it is possible to provide a model generation device, a conversion device, a model generation method, and a program that enable high-resolution images to be obtained regardless of the antenna configuration of the radar.

[0013] FIG. 1 is a diagram illustrating an overview of a model generation device according to a first embodiment. FIG. 2 is a diagram illustrating a model generated by the model generation device according to the first embodiment. FIG. 3 is a diagram for explaining a first image and a second image according to the first embodiment. FIG. 4 is a diagram illustrating an antenna arrangement in one subset. FIG. 5 is a diagram illustrating a first modified example of an antenna arrangement in one subset. FIG. 6 is a diagram illustrating a second modified example of an antenna arrangement in one subset. FIG. 7 is a diagram illustrating a functional configuration of a model generation device according to the first embodiment. FIG. 8 is a diagram illustrating a computer for realizing the model generation device. FIG. 9 is a diagram illustrating an overview of a model generation method according to the first embodiment. FIG. 10 is a diagram illustrating a functional configuration of a model generation device according to Modification 1. FIG. 11 is a diagram for explaining a first image and a second image according to a second embodiment. FIG. 12 is a diagram illustrating a model generated by a model generation device according to a third embodiment. FIG. 13 is a diagram illustrating an overview of a conversion device according to a fourth embodiment. FIG. 14 is a diagram illustrating a functional configuration of a conversion device according to the fourth embodiment. FIG. 15 is a diagram illustrating an overview of a conversion method according to the fourth embodiment.

[0014] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In all the drawings, like components are designated by like reference numerals, and the description thereof will be omitted as appropriate.

[0015] First Embodiment FIG. 1 is a diagram illustrating an overview of a model generation device 10 according to a first embodiment. The model generation device 10 includes a learning unit 130. The learning unit 130 generates a model by performing machine learning using a first image and a second image. The first image is an image generated using first radar signal data. The second image is an image generated using second radar signal data. The first radar signal data is a signal extracted from the second radar signal data.

[0016] According to this model generating device 10, a model that enables high-resolution images to be obtained regardless of the antenna configuration of the radar can be obtained.

[0017] A detailed example of the model generating device 10 will be described below.

[0018] The resolution of an image obtained by a radar that transmits and receives radio waves using multiple antennas usually depends on the antenna configuration. Specifically, the higher the frequency f of the radio waves transmitted and received by the antennas (or the wider the frequency band), the higher the resolution of the image obtained. Furthermore, the more antennas included in the radar, the higher the resolution of the image obtained. Note that in a radar, multiple antennas are usually arranged at a predetermined interval (e.g., λ / 2) based on the wavelength λ of the radio waves transmitted and received by the antennas. Therefore, for a fixed aperture size, the higher the frequency f, the greater the number of antennas.

[0019] For example, if the aperture size of a radar is reduced to make it smaller, the number of antennas that can be included in the radar will be reduced. Another possibility is to reduce the number of antennas to lower the manufacturing cost of the radar. Therefore, a technology that enables high-resolution images to be obtained even when a radar with a small number of antennas is used is useful.

[0020] FIG. 2 is a diagram illustrating a model 30 generated by the model generation device 10 according to this embodiment. The model generation device 10 according to this embodiment is a device that generates a model 30 that can increase the resolution of a radar image. Note that a radar image is an image obtained by processing radar signal data obtained by a radar. Existing imaging processes can be used for the process of obtaining a radar image from radar signal data. An image is input to the model 30 as input data. The image input to the model 30 is also referred to as an input image. The input image is a radar image. An image is output from the model 30 as output data. The image output from the model 30 is also referred to as an output image. By using the model 30, an output image with a higher resolution than the input image can be obtained.

[0021] The model 30 includes a neural network. The model generation device 10 according to this embodiment generates the model 30 by performing machine learning using training data that combines a first image and a second image. Here, the second image has a higher resolution than the first image. In the machine learning, the model generation device 10 uses the first image as input data to the model and the second image as ground truth data. However, the model 30 may be generated by a method other than a neural network.

[0022] 3 is a diagram illustrating a first image 61 and a second image 62 according to this embodiment. The first image 61 and the second image 62 are generated using a second radar 52. Here, the second radar 52 includes the first radar 51 as a part of the second radar 52. In the example of FIG. 3 , the second radar 52 includes a plurality of subsets 50. In the example of FIG. 3 , the first radar 51 is one of the plurality of subsets 50 included in the second radar 52.

[0023] However, the configurations of the second radar 52 and the first radar 51 and the relationship between the second radar 52 and the first radar 51 are not limited to the example in Fig. 3. For example, in the example in Fig. 3, the first radar 51 is a subset 50 located in the center of the second radar 52, but the first radar 51 may be a subset 50 located in another position. Also, in the example in Fig. 3, the first radar 51 consists of one subset 50, but the first radar 51 may be composed of two or more subsets 50. However, the first radar 51 does not include all of the subsets 50 included in the second radar 52.

[0024] FIG. 4 is a diagram illustrating an example of antenna arrangement in one subset 50. In FIG. 4, transmitting antennas 501 are indicated by white circles, and receiving antennas 502 are indicated by black circles. The subset 50 includes multiple transmitting antennas 501 and multiple receiving antennas 502. The transmitting antennas 501 and receiving antennas 502 are collectively referred to as antennas. In the subset 50, the multiple antennas are arranged to form a rectangle. The multiple antennas are arranged so that one pair of opposing sides of the rectangle is represented by a row of transmitting antennas 501, and the other pair of opposing sides is represented by a row of receiving antennas 502.

[0025] The first radar 51 and the second radar 52 transmit and receive radio waves in a predetermined frequency band. The first radar 51 and the second radar 52 may be, for example, Stepped Frequency Continuous Wave (SFCW) radars. Alternatively, the first radar 51 and the second radar 52 may be Frequency Modulated Continuous Wave (FMCW) radars. The frequency of the radio waves transmitted by the transmitting antenna 501 changes stepwise or continuously within the predetermined frequency band. In the subset 50, the distance between adjacent antennas is, for example, λ / 2, but is not limited to λ / 2. λ is the wavelength corresponding to the highest frequency in the frequency band of the radio waves transmitted and received by the first radar 51 and the second radar 52. Note that the antenna arrangement in the subset 50 is not limited to the example shown in FIG. 4 .

[0026] The second radar 52 generates second radar signal data by transmitting and receiving radio waves. The second radar signal data includes multiple pieces of reception data based on the reception signals of the receiving antennas 502. Specifically, the multiple transmitting antennas 501 transmit radio waves at different times. All receiving antennas 502 included in the second radar 52 receive the radio waves transmitted from each transmitting antenna 501 and reflected by an object. A receiving antenna 502 may also receive radio waves transmitted from a transmitting antenna 501 included in a subset 50 different from that receiving antenna 502 and reflected by an object. In this way, reception data is obtained for all pairs of transmitting antennas 501 and receiving antennas 502 included in the second radar 52. The radar signal data of the second radar 52 includes all of this reception data.

[0027] 5 is a diagram showing a first modified example of antenna arrangement in one subset 50. In the first modified example, transmitting / receiving antennas 503 having both transmitting and receiving functions are arranged at the four corners. However, the transmitting and receiving functions of the transmitting / receiving antennas 503 are not operated simultaneously. While the transmitting / receiving antenna 503 is functioning as a receiving antenna, the transmitting / receiving antenna 503 does not function as a transmitting antenna. While the transmitting / receiving antenna 503 is functioning as a transmitting antenna, the transmitting / receiving antenna 503 does not function as a receiving antenna.

[0028] 6 is a diagram showing a second modified example of the antenna arrangement in one subset 50. In the second modified example, no antennas are arranged at the four corners.

[0029] 4 to 6, the direction in which the antennas are arranged is not particularly limited. The receiving antennas 502 may be arranged vertically and the transmitting antennas 501 may be arranged horizontally. Furthermore, the number of receiving antennas 502 and transmitting antennas 501 is not limited to the examples in FIGS. 4 to 6.

[0030] Returning to Fig. 3 , the first image 61 and the second image 62 will be further described. As described above, the second radar signal data is signal data obtained by the second radar 52, which includes the first radar 51. The first radar signal data is signal data obtained by extracting data corresponding to the signal data of the first radar 51 from the second radar signal data. It can be said that the second radar signal data is data measured simultaneously with the first radar signal data.

[0031] Specifically, received data obtained by receiving radio waves transmitted from a transmitting antenna 501 included in the first radar 51 and reflected by an object using a receiving antenna 502 included in the first radar 51 is extracted from the second radar signal data. First radar signal data including the extracted multiple pieces of received data is then generated. The first radar signal data does not include received data based on radio waves transmitted by a transmitting antenna 501 not included in the first radar 51. Furthermore, the first radar signal data does not include received data based on radio waves received by a receiving antenna 502 not included in the first radar 51.

[0032] The first radar signal data includes signal data for all or only a part of the frequency bands of the radio waves transmitted and received by the second radar. That is, data for all frequency bands may be extracted from the second radar signal data and used as the first radar signal data, or data for only a part of the frequency bands may be extracted and used as the first radar signal data. In the latter case, data other than the extracted part of the frequency bands is not included in the first radar signal data. In this way, the frequency band of the second radar signal data may be wider than the frequency band of the first radar signal data. That is, the frequency band of the radio waves transmitted and received by the first radar 51 may virtually be narrower than the frequency band of the radio waves transmitted and received by the second radar 52.

[0033] The first image 61 is an image generated by performing imaging processing on the first radar signal data.

[0034] Similarly, the second radar signal data is subjected to imaging processing to obtain an image 63. Because the aperture 520 of the second radar 52 is larger than the aperture 510 of the first radar 51, the image 63 has a larger angle of view than the angle of view of the first image 61. Furthermore, because the number of antennas included in the second radar 52 is greater than the number of antennas included in the first radar 51, the resolution of the image 63 is higher than the resolution of the first image 61.

[0035] The second image 62 is an image obtained by cutting out a portion of the image 63 generated using the second radar signal data. Specifically, the second image 62 is an image obtained by cutting out a region corresponding to the first image 61 from the radar image 63 generated using the second radar signal data. The position and size of the second image 62 cut out from the image 63 are determined based on the positional and dimensional relationship between the aperture 510 of the first radar 51 and the aperture 520 of the second radar 52. In this way, the first image 61 and the second image 62, which capture the same target region, can be obtained. Furthermore, the second image 62 can be obtained with a higher resolution than the first image 61.

[0036] The training data includes a first image 61 and a second image 62 obtained from the same second radar signal data.

[0037] By using such training data, a model 30 can be obtained that can increase the resolution of radar images obtained with a radar having a small number of antennas.

[0038] 7 is a diagram illustrating an example of the functional configuration of the model generation device 10 according to this embodiment. In the example of FIG. 7, the model generation device 10 further includes an acquisition unit 110, a first storage unit 81, and a model storage unit 82. However, the first storage unit 81 may be provided outside the model generation device 10. Furthermore, the model storage unit 82 may be provided outside the model generation device 10.

[0039] The first storage unit 81 stores, for example, training data that combines the first image 61 and the second image 62. The training data can be prepared in advance and stored in the first storage unit 81.

[0040] The model storage unit 82 holds information for configuring the model 30. The information for configuring the model 30 includes a plurality of parameters of the neural network.

[0041] The acquisition unit 110 of the model generation device 10 reads and acquires the training data, i.e., the set of the first image 61 and the second image 62, from the first storage unit 81. However, the acquisition unit 110 may acquire the training data from another device. In that case, the first storage unit 81 may not be provided.

[0042] The learning unit 130 updates the parameters stored in the model storage unit 82 by performing machine learning using the training data acquired by the acquisition unit 110. Machine learning can be performed using existing technology. For example, the learning unit 130 inputs a first image 61 to the model 30 and obtains an output image from the model 30. Then, the learning unit 130 calculates a loss using the output image and a second image 62 that is included in the same training data as the first image 61 input to the model 30. Then, the learning unit 130 updates the parameters of the model 30 so as to reduce the loss.

[0043] The first storage unit 81 can store multiple pieces of training data. The acquisition unit 110 can acquire the multiple pieces of training data. The learning unit 130 repeatedly updates the parameters using the multiple pieces of training data. For example, the learning unit 130 repeats updating the parameters until a predetermined termination condition is satisfied. The predetermined termination condition is, for example, at least one of the number of parameter updates reaching a predetermined number and the loss becoming smaller than a predetermined standard.

[0044] The hardware configuration of the model generating device 10 is described below. Each functional component of the model generating device 10 (the acquisition unit 110 and the learning unit 130) may be realized by hardware that realizes each functional component (e.g., a hardwired electronic circuit, etc.), or by a combination of hardware and software (e.g., a combination of an electronic circuit and a program that controls it). A case in which each functional component of the model generating device 10 is realized by a combination of hardware and software will be further described below.

[0045] FIG. 8 is a diagram illustrating a computer 1000 for implementing the model generation device 10. The computer 1000 may be any computer. For example, the computer 1000 may be a system on chip (SoC), a personal computer (PC), a server machine, a tablet terminal, or a smartphone. The computer 1000 may be a dedicated computer designed to implement the model generation device 10, or may be a general-purpose computer. The model generation device 10 may be implemented by a single computer 1000 or by a combination of multiple computers 1000.

[0046] The computer 1000 includes a bus 1020, a processor 1040, a memory 1060, a storage device 1080, an input / output interface 1100, and a network interface 1120. The bus 1020 is a data transmission path through which the processor 1040, the memory 1060, the storage device 1080, the input / output interface 1100, and the network interface 1120 transmit and receive data to and from each other. However, the method of interconnecting the processor 1040 and other components is not limited to bus connection. The processor 1040 may be any of various processors, such as a central processing unit (CPU), a graphics processing unit (GPU), or a field-programmable gate array (FPGA). The memory 1060 is a main storage device implemented using a random access memory (RAM) or the like. The storage device 1080 is an auxiliary storage device implemented using a hard disk, a solid state drive (SSD), a memory card, a read-only memory (ROM), or the like.

[0047] The input / output interface 1100 is an interface for connecting the computer 1000 to an input / output device. For example, an input device such as a keyboard and an output device such as a display are connected to the input / output interface 1100. The input / output interface 1100 may be connected to the input device or output device via a wireless connection or a wired connection.

[0048] The network interface 1120 is an interface for connecting the computer 1000 to a network. This communication network may be, for example, a LAN (Local Area Network) or a WAN (Wide Area Network). The network interface 1120 may be connected to the network wirelessly or by wire.

[0049] The storage device 1080 stores program modules that realize each functional component of the model generating device 10. The processor 1040 reads each of these program modules into the memory 1060 and executes them to realize the function corresponding to each program module.

[0050] Furthermore, when the first storage unit 81 is provided inside the model generation device 10, the first storage unit 81 is realized, for example, by using the storage device 1080. When the model storage unit 82 is provided inside the model generation device 10, the model storage unit 82 is realized, for example, by using the storage device 1080.

[0051] 9 is a diagram showing an overview of a model generation method according to this embodiment. In the model generation method according to this embodiment, one or more computers generate a model 30 by performing machine learning using a first image 61 and a second image 62 (S11). The first image 61 is an image generated using first radar signal data. The second image 62 is an image generated using second radar signal data. The first radar signal data is a signal extracted from the second radar signal data.

[0052] As described above, according to this embodiment, the learning unit 130 generates a model by performing machine learning using the first image 61 and the second image 62. Therefore, a model 30 is obtained that enables obtaining a high-resolution image regardless of the antenna configuration of the radar.

[0053] 10 is a diagram illustrating a functional configuration of a model generation device 10 according to Modification 1. The model generation device 10 according to this modification is the same as the model generation device 10 according to the first embodiment, except for the points described below.

[0054] In this modification, the model generation device 10 includes a training data generation unit 120. A first storage unit 81 stores data for generating training data in advance. That is, the first storage unit 81 stores data for obtaining a first image 61 and a second image 62 in advance. An acquisition unit 110 according to this modification reads and acquires the data for generating training data from the first storage unit 81. The training data generation unit 120 then generates training data (a set of the first image 61 and the second image 62) using the data acquired by the acquisition unit 110. A learning unit 130 generates a model 30 in the same manner as in the first embodiment, using the training data generated by the training data generation unit 120.

[0055] The data for generating the training data is at least one of the second radar signal data, the first radar signal data, and the image 63. That is, the data for generating the training data may be only the second radar signal data, a combination of the second radar signal data and the first radar signal data, or a combination of the second radar signal data and the image 63. The data for generating the training data may be a combination of the first radar signal data and the image 63. Alternatively, the data for generating the training data may be a combination of one of the first image 61 and the second image 62 and data for generating the other of the first image 61 and the second image 62.

[0056] As the second radar signal data, data obtained from the second radar 52 can be stored in advance in the first storage unit 81. The first radar signal data and the image 63 can be generated and prepared by another information processing device or the like and stored in advance in the first storage unit 81.

[0057] However, the acquisition unit 110 may acquire data for generating training data from a radar or another information processing device, in which case the first storage unit 81 may not be provided.

[0058] The acquisition unit 110 and the training data generation unit 120 according to this modification will be described in detail below.

[0059] The acquisition unit 110 acquires data for generating training data.

[0060] When the acquisition unit 110 acquires the second radar signal data, the training data generation unit 120 generates at least one of the first image 61 and the second image 62 using the second radar signal data.

[0061] Specifically, the training data generation unit 120 extracts the first radar signal data from the second radar signal data as described above, and then performs imaging processing on the first radar signal data to generate the first image 61.

[0062] Furthermore, the training data generation unit 120 generates an image 63 by performing imaging processing on the second radar signal data, and cuts out a second image 62 from the image 63. In this manner, the second image 62 can be generated.

[0063] When the acquisition unit 110 acquires the first radar signal data, the training data generation unit 120 generates the first image 61 by performing imaging processing on the first radar signal data.

[0064] When the acquisition unit 110 acquires the image 63 , the training data generation unit 120 cuts out the second image 62 from the image 63 to generate the second image 62 .

[0065] In this way, the training data generator 120 can generate training data.

[0066] The first storage unit 81 can store multiple pieces of data for generating training data. The acquisition unit 110 can acquire multiple pieces of data for generating training data. The training data generation unit 120 generates multiple pieces of training data. As in the first embodiment, the learning unit 130 repeatedly updates the parameters using the multiple pieces of training data.

[0067] The hardware configuration of a computer that realizes the model generation device 10 according to this modification is, for example, represented in Fig. 8 , similar to the model generation device 10. However, a program module that realizes the function of the training data generation unit 120 is further stored in the storage device 1080 of the computer 1000 that realizes the model generation device 10 according to this modification.

[0068] According to this modification, the same actions and effects as those of the first embodiment can be obtained.

[0069] Second Embodiment Similar to the first embodiment, an overview of a model generation device 10 according to a second embodiment is shown in Fig. 1. Fig. 11 is a diagram for explaining a first image 61 and a second image 62 according to this embodiment. The model generation device 10 according to this embodiment is the same as the model generation device 10 according to the first embodiment except for the points described below.

[0070] The model generating device 10 according to this embodiment includes a learning unit 130. The learning unit 130 generates the model 30 by performing machine learning using a first image 61 and a second image 62. The first image 61 is an image generated using first radar signal data obtained by the first radar 51. The second image 62 is an image generated using second radar signal data obtained by the second radar 52. In this embodiment, the center frequency of the frequency band of radio waves transmitted and received by the second radar 52 is higher than the center frequency of the frequency band of radio waves transmitted and received by the first radar 51. This will be described in detail below.

[0071] In the present embodiment, the first radar 51 is not included in the second radar 52. In the example of Fig. 11 , the aperture 510 of the first radar 51 and the aperture 520 of the second radar 52 are provided independently of each other. In the present embodiment, the size of the aperture 520 of the second radar 52 may be the same as the size of the aperture 510 of the first radar 51. In this case, the angle of view of the first image 61 and the angle of view of the second image 62 are the same.

[0072] The number of antennas included in the second radar 52 may be greater than the number of antennas included in the first radar 51. The aperture 510 of the first radar 51 may at least partially overlap with the aperture 520 of the second radar 52. In other words, part of the antenna of the first radar 51 may also serve as the antenna of the second radar 52.

[0073] In this embodiment, the radio waves transmitted and received by the second radar 52 have a higher frequency than the radio waves transmitted and received by the first radar 51. Therefore, the second image 62 has a higher resolution than the first image 61. Specifically, as described above, the center frequency of the frequency band of the radio waves transmitted and received by the second radar 52 is higher than the center frequency of the frequency band of the radio waves transmitted and received by the first radar 51. Note that the center frequency is the arithmetic mean of the highest and lowest frequencies in the frequency band.

[0074] As long as the second image 62 has a higher resolution than the first image 61, there is no particular limitation on the relationship between the bandwidth of the frequency band of the radio waves transmitted and received by the first radar 51 and the bandwidth of the frequency band of the radio waves transmitted and received by the second radar 52. That is, the bandwidth of the frequency band of the radio waves transmitted and received by the first radar 51 may be wider or narrower than the bandwidth of the frequency band of the radio waves transmitted and received by the second radar 52. Alternatively, the bandwidth of the frequency band of the radio waves transmitted and received by the first radar 51 may be the same as the bandwidth of the frequency band of the radio waves transmitted and received by the second radar 52.

[0075] The first image 61 is generated by performing imaging processing on the first radar signal data obtained by the first radar 51. The second image 62 is generated by performing imaging processing on the second radar signal data obtained by the second radar 52.

[0076] The first radar signal data and the second radar signal data are obtained by measuring substantially the same target (area) using the first radar 51 and the second radar 52 under substantially the same conditions. This results in a first image 61 and a second image 62 that have a corresponding correlation with each other. For example, the first image 61 and the second image 62 can be obtained by capturing substantially the same target area. For example, the first radar 51 and the second radar 52 can be mounted on the same board, and measurement conditions, etc. can be set and controlled. Note that, to avoid interference, it is preferable to slightly stagger the measurement timings of the first radar 51 and the second radar 52. Alternatively, it is preferable to remove frequency components that cause interference from the signals received by the receiving antenna 502 using a filter or the like. That is, the frequency components of the second radar 52 are removed from the signals received by the first radar 51, and the frequency components of the first radar 51 are removed from the signals received by the second radar 52. In this case, it is preferable that the frequency bands of the first radar 51 and the second radar 52 do not overlap.

[0077] The training data includes a first image 61 and a second image 62 that have a corresponding correlation with each other. That is, the training data includes the first image 61 and the second image 62 that are obtained by measuring approximately the same object (area) under approximately the same conditions using the first radar 51 and the second radar 52.

[0078] By using such training data, a model 30 can be obtained that can increase the resolution of radar images obtained with a radar having a small number of antennas.

[0079] The functional configuration of the model generating device 10 according to this embodiment is illustrated in Fig. 7 , similarly to the first embodiment. As in the first embodiment, the acquisition unit 110 reads and acquires training data, i.e., a set of a first image 61 and a second image 62, from the first storage unit 81. The learning unit 130 performs machine learning using the training data acquired by the acquisition unit 110, thereby updating the parameters of the model 30 stored in the model storage unit 82.

[0080] As in the first embodiment, the first storage unit 81 can store a plurality of sets of training data. The acquisition unit 110 can acquire the plurality of sets of training data. The learning unit 130 repeatedly updates the parameters using the plurality of sets of training data.

[0081] An overview of the model generation method according to this embodiment is shown in Fig. 9 , similarly to the first embodiment. In the model generation method according to this embodiment, one or more computers generate a model by performing machine learning using a first image 61 and a second image 62 (S11). The first image 61 is an image generated using first radar signal data obtained by the first radar 51. The second image 62 is an image generated using second radar signal data obtained by the second radar 52. The center frequency of the frequency band of radio waves transmitted and received by the second radar 52 is higher than the center frequency of the frequency band of radio waves transmitted and received by the first radar 51.

[0082] According to this embodiment, the same actions and effects as those of the first embodiment can be obtained.

[0083] (Modification 2) The functional configuration of the model generation device 10 according to Modification 2 is illustrated in Fig. 10, similarly to Modification 1. The model generation device 10 according to this modification is the same as the model generation device 10 according to the second embodiment, except for the points described below.

[0084] In this modification, the model generation device 10 includes a training data generation unit 120. A first storage unit 81 stores data for generating training data in advance. That is, the first storage unit 81 stores data for obtaining a first image 61 and a second image 62 in advance. An acquisition unit 110 according to this modification reads and acquires data for generating training data from the first storage unit 81. Then, the training data generation unit 120 generates training data using the data acquired by the acquisition unit 110. A learning unit 130 generates a model 30 using the training data generated by the training data generation unit 120, as in the second embodiment.

[0085] The data for generating the training data is the first radar signal data and the second radar signal data, or may be a combination of one of the first image 61 and the second image 62 and the data for generating the other of the first image 61 and the second image 62.

[0086] As the first radar signal data, data obtained from the first radar 51 can be stored in advance in the first storage unit 81. As the second radar signal data, data obtained from the second radar 52 can be stored in advance in the first storage unit 81.

[0087] However, the acquisition unit 110 may acquire data for generating training data directly from the radar, in which case the first storage unit 81 may not be provided.

[0088] The acquisition unit 110 and the training data generation unit 120 according to this modification will be described in detail below.

[0089] When the acquisition unit 110 acquires first radar signal data, the training data generation unit 120 performs imaging processing on the first radar signal data to generate a first image 61. When the acquisition unit 110 acquires second radar signal data, the training data generation unit 120 performs imaging processing on the second radar signal data to generate a second image 62.

[0090] In this way, the training data generator 120 can generate training data.

[0091] The first storage unit 81 can store multiple pieces of data for generating training data. The acquisition unit 110 can acquire multiple pieces of data for generating training data. The training data generation unit 120 generates multiple pieces of training data. As in the second embodiment, the learning unit 130 repeatedly updates the parameters using the multiple pieces of training data.

[0092] The hardware configuration of a computer that realizes the model generation device 10 according to this modification is, for example, represented in Fig. 8 , similar to the model generation device 10. However, a program module that realizes the function of the training data generation unit 120 is further stored in the storage device 1080 of the computer 1000 that realizes the model generation device 10 according to this modification.

[0093] According to this modification, the same actions and effects as those of the first embodiment can be obtained.

[0094] 12 is a diagram illustrating a model 30 generated by a model generation device 10 according to a third embodiment. The model generation device 10 according to this embodiment is the same as the model generation device 10 according to at least one of the first and second embodiments, modified example 1, and modified example 2, except for the points described below.

[0095] The learning unit 130 according to this embodiment generates the model 30 by performing machine learning using a plurality of first images 61, which are a plurality of video frames, and a plurality of second images 62, which are also a plurality of video frames. This will be described in detail below.

[0096] In this embodiment, a video is input to the model 30 as input data. The model 30 outputs a video as output data. In other words, the model 30 receives a plurality of time-series images, i.e., a plurality of video frames, as input data. The video input to the model 30 is also referred to as an input video. The input video is a video made up of radar images. The model 30 outputs a plurality of time-series images, i.e., a plurality of video frames, as output data. The video output from the model 30 is also referred to as an output video. By using the model 30, an output video with a higher resolution than the input video can be obtained. In the model 30 according to this embodiment, the resolution enhancement of each image is influenced by the images before and after that image. Therefore, the resolution enhancement of the video can be performed with greater accuracy.

[0097] In this embodiment, the model 30 also includes a neural network. The model generation device 10 according to this embodiment generates the model 30 by performing machine learning using training data that combines a first video and a second video. The first video is a video made up of a plurality of first images 61. The second video is a video made up of a plurality of second images 62. In the machine learning, the model generation device 10 uses the first video as input data to the model and the second video as ground truth data.

[0098] Each first image 61 and each second image 62 according to this embodiment is as described in the first or second embodiment. By repeatedly performing measurements using the first radar 51 and the second radar 52, a plurality of first images 61 and a plurality of second images 62 are obtained. In other words, a first video and a second video are obtained.

[0099] The functional configuration of the model generating device 10 according to this embodiment is illustrated in Fig. 7, similar to the first and second embodiments. Alternatively, the functional configuration of the model generating device 10 according to this embodiment is illustrated in Fig. 10, similar to the first and second modifications.

[0100] The acquisition unit 110 of the model generation device 10 according to this embodiment acquires training data in the same manner as in the first and second embodiments. Alternatively, the acquisition unit 110 acquires data for generating training data, and the training data generation unit 120 generates the training data in the same manner as in the first and second modifications.

[0101] The learning unit 130 updates the parameters stored in the model storage unit 82 by performing machine learning using training data. Machine learning can be performed using existing technology. For example, the learning unit 130 inputs a first video into the model 30 and obtains an output video from the model 30. Then, the learning unit 130 calculates a loss using the output video and a second video included in the same training data as the first video input to the model 30. The learning unit 130 then updates the parameters of the model 30 so as to reduce the loss. Furthermore, the learning unit 130 can repeatedly update the parameters using multiple training data.

[0102] Next, the operation and effect of this embodiment will be described. In this embodiment, the same operation and effect as in the first embodiment can be obtained. In addition, a model 30 can be obtained that can obtain high-resolution video regardless of the antenna configuration of the radar.

[0103] (Fourth embodiment) Fig. 13 is a diagram showing an overview of a conversion device 20 according to the fourth embodiment. The conversion device 20 according to this embodiment includes a conversion unit 230. The conversion unit 230 converts a target image using a model 30. The model 30 used by the conversion unit 230 is the model 30 generated by the model generation device 10 according to at least one of the first to third embodiments, Modification 1, and Modification 2. This will be described in detail below.

[0104] 14 is a diagram illustrating an example of the functional configuration of the conversion device 20 according to this embodiment. In the example of FIG. 14, the conversion device 20 further includes a target data acquisition unit 210, a second storage unit 83, and an output data storage unit 84. However, the second storage unit 83 may be provided outside the conversion device 20. The output data storage unit 84 may be provided outside the conversion device 20.

[0105] The second storage unit 83 stores, for example, a target image. The target image is, for example, a radar image obtained by the third radar. The third radar preferably has the same configuration (number of antennas, antenna arrangement, etc.) as the first radar 51. The third radar may be the first radar 51 itself, or may be a radar different from the first radar 51. The target image is an image obtained by performing imaging processing on the third laser signal data obtained by the third radar. Note that the measurement conditions (frequency band, measurement time, etc.) when obtaining the third radar signal data by the third radar are preferably the same as the measurement conditions for obtaining the first radar signal data by the first radar 51. The frequency band of the third radar signal data is preferably the same as the frequency band of the first radar signal data.

[0106] The second storage unit 83 may store data for generating the target image instead of storing the target image, such as third radar signal data.

[0107] The target data acquisition unit 210 of the conversion device 20 acquires the target image or data for generating the target image from the second storage unit 83. However, the target data acquisition unit 210 may acquire the target image or data for generating the target image from a third radar or another information processing device. In this case, the second storage unit 83 may not be provided.

[0108] When the target data acquisition unit 210 acquires data for generating a target image, the conversion unit 230 performs imaging processing on the data to generate the target image.

[0109] The conversion unit 230 inputs a target image to the model 30. The conversion unit 230 can access the model storage unit 82 and can use the model 30 stored in the model storage unit 82. The model storage unit 82 may be provided within the conversion device 20. The conversion unit 230 can obtain an output image of the model 30. The conversion unit 230 can obtain, as the output image, an image with a higher resolution than the input target image.

[0110] As described in the third embodiment, when the model 30 is capable of receiving input of a moving image, the conversion unit 230 may obtain a plurality of target images, which are a plurality of moving image frames, and input them to the model 30. In this case, the conversion unit 230 can obtain, as an output moving image, a moving image with a higher resolution than the input moving image.

[0111] The conversion section 230 stores the obtained output image or output video in, for example, the output data storage section 84. Alternatively, the conversion section 230 may output the output image or output video to another device.

[0112] For example, a high-resolution image can be obtained by converting a target image obtained by a radar with a small aperture size and a small number of antennas for miniaturization using the conversion device 20. Also, a high-resolution image can be obtained by converting a target image obtained by a radar with a small number of antennas for cost reduction using the conversion device 20.

[0113] The hardware configuration of the computer that realizes the conversion device 20 is shown in Fig. 8, for example, similar to the model generation device 10. However, the storage device 1080 of the computer 1000 that realizes the conversion device 20 of this embodiment stores program modules that realize the functions of each functional component (target data acquisition unit 210 and conversion unit 230) of the conversion device 20 of this embodiment.

[0114] Furthermore, when the second storage unit 83 is provided inside the conversion device 20, the second storage unit 83 is realized, for example, by using the storage device 1080 of the computer 1000 that realizes the conversion device 20. When the output data storage unit 84 is provided inside the conversion device 20, the output data storage unit 84 is realized, for example, by using the storage device 1080 of the computer 1000 that realizes the conversion device 20. When the model storage unit 82 is provided inside the conversion device 20, the model storage unit 82 is realized, for example, by using the storage device 1080 of the computer 1000 that realizes the conversion device 20.

[0115] 15 is a diagram showing an overview of the conversion method according to this embodiment. In the conversion method according to this embodiment, one or more computers convert a target image using a model 30 (S21). The model 30 is generated by the model generation device 10 according to at least one of the first to third embodiments, Modification 1, and Modification 2.

[0116] As described above, according to this embodiment, the conversion unit 230 converts the target image using the model 30. Therefore, the target image can be converted into an image with high resolution.

[0117] Although the embodiments of the present invention have been described above with reference to the drawings, these are merely examples of the present invention, and various other configurations can also be adopted.

[0118] The order of steps performed in each embodiment is not limited to the order described. In each embodiment, the order of steps can be changed as long as it does not cause any problems. Furthermore, the above-described embodiments can be combined as long as the content is not contradictory.

[0119] Some or all of the above embodiments can be described as, but are not limited to, the following supplementary notes. 1-1. A model generation device comprising: learning means for generating a model by performing machine learning using a first image and a second image, wherein the first image is an image generated using first radar signal data, the second image is an image generated using second radar signal data, and the first radar signal data is a signal extracted from the second radar signal data. 1-2. A model generation device according to 1-1, wherein the second radar signal data is signal data obtained by a second radar including a first radar, and the first radar signal data is signal data obtained by extracting data equivalent to the signal data of the first radar from the second radar signal data. 1-3. A model generation device according to 1-2, wherein the aperture of the second radar is larger than the aperture of the first radar. 1-4. A model generation device according to 1-2 or 1-3. 1-5. The model generation device described in any one of 1-1. to 1-4., wherein the first radar signal data includes signal data for all or only a part of the frequency bands of radio waves transmitted and received by the second radar. 1-6. The model generation device described in 1-5., wherein the second image is an image obtained by cutting out a part of an image generated using the second radar signal data. 1-7. The model generation device described in any one of 1-1. to 1-6., wherein the second image is an image obtained by cutting out a region corresponding to the first image from an image generated using the second radar signal data. 1-8. The model generation device described in any one of 1-1. to 1-7., wherein the learning means generates a model by performing machine learning using a plurality of the first images, which are a plurality of video frames, and a plurality of the second images, which are also a plurality of video frames.2-1. A model generation device comprising: a learning means for generating a model by performing machine learning using a first image and a second image; the first image is an image generated using first radar signal data obtained by a first radar; the second image is an image generated using second radar signal data obtained by a second radar; and the center frequency of the frequency band of radio waves transmitted and received by the second radar is higher than the center frequency of the frequency band of radio waves transmitted and received by the first radar. 2-2. The model generation device described in 2-1., wherein the aperture size of the second radar is the same as the aperture size of the first radar. 2-3. The model generation device described in 2-1. or 2-2., wherein the second image has a higher resolution than the first image. 2-4. The model generation device described in any one of 2-1. to 2-3., wherein the learning means generates a model by performing machine learning using a plurality of the first images, which are a plurality of video frames, and a plurality of the second images, which are also a plurality of video frames. 3-1. 1-1. to 1-8. 3-1. A conversion device comprising a conversion means for converting a target image using a model generated by the model generation device described in any one of 1-1. to 1-4. 3-2. A conversion device comprising a conversion means for converting a target image using a model generated by the model generation device described in any one of 1-2. to 1-4. 3-3. A conversion device comprising a conversion means for converting a target image using a model generated by the model generation device described in any one of 2-1. to 2-4. 3-4. A conversion device according to 3-2. or 3-3., wherein the target image is an image obtained by a radar having the same configuration as the first radar. 4-1. A model generation method, wherein one or more computers perform machine learning using a first image and a second image, to generate a model, wherein the first image is an image generated using first radar signal data, the second image is an image generated using second radar signal data, and the first radar signal data is a signal extracted from the second radar signal data.4-2. A model generation method according to 4-1., wherein the second radar signal data is signal data obtained by a second radar including a first radar, and the first radar signal data is signal data obtained by extracting data equivalent to the signal data of the first radar from the second radar signal data. 4-3. A model generation method according to 4-2., wherein the aperture of the second radar is larger than the aperture of the first radar. 4-4. A model generation method according to 4-2. or 4-3., wherein the first radar signal data includes signal data for all or only a part of the frequency bands of radio waves transmitted and received by the second radar. 4-5. A model generation method according to any one of 4-1. to 4-4., wherein the second image is an image obtained by cutting out a part of an image generated using the second radar signal data. 4-6. A model generation method according to 4-5. 5-1. A model generation method in which one or more computers generate a model by performing machine learning using a first image and a second image, wherein the first image is an image generated using first radar signal data obtained by extracting a region corresponding to the first image from an image generated using the second radar signal data. 4-7. A model generation method in which the second image is an image obtained by extracting a region corresponding to the first image from an image generated using the second radar signal data. 4-8. A model generation method in which the one or more computers generate the model by performing machine learning using a plurality of the first images, which are a plurality of video frames, and a plurality of the second images, which are also a plurality of video frames. 5-1. A model generation method in which one or more computers generate a model by performing machine learning using a first image and a second image, wherein the first image is an image generated using first radar signal data obtained by a first radar, and the second image is an image generated using second radar signal data obtained by a second radar, and the center frequency of the frequency band of the radio waves transmitted and received by the second radar is higher than the center frequency of the frequency band of the radio waves transmitted and received by the first radar.5-2. A model generation method according to 5-1, wherein the aperture size of the second radar is the same as the aperture size of the first radar. 5-3. A model generation method according to 5-1 or 5-2, wherein the second image has a higher resolution than the first image. 5-4. A model generation method according to any one of 5-1 to 5-3, wherein the one or more computers generate the model by performing machine learning using the first images, which are video frames, and the second images, which are video frames. 6-1. A conversion method in which one or more computers convert a target image using a model generated by a model generation device according to any one of 1-1 to 1-8. 6-2. A conversion method in which one or more computers convert a target image using a model generated by a model generation device according to any one of 1-2 to 1-4. 6-3. A conversion method in which one or more computers convert a target image using a model generated by a model generation device according to any one of 2-1 to 2-4. 6-4. A conversion method for converting a target image using a model generated by the model generation device described in any one of 6-2. and 6-3.. 6-4. A conversion method according to 6-2. or 6-3., wherein the target image is an image obtained by a radar having the same configuration as the first radar. 7-1. A program for causing a computer to function as a learning means for generating a model by performing machine learning using a first image and a second image, wherein the first image is an image generated using first radar signal data, the second image is an image generated using second radar signal data, and the first radar signal data is a signal extracted from the second radar signal data. 7-2. A program according to 7-1., wherein the second radar signal data is signal data obtained by a second radar including the first radar, and the first radar signal data is signal data obtained by extracting data equivalent to the signal data of the first radar from the second radar signal data. 7-3. A program according to 7-2., wherein the aperture of the second radar is larger than the aperture of the first radar.7-4. A program according to 7-2. or 7-3., wherein the first radar signal data includes signal data for all or only a part of the frequency bands of radio waves transmitted and received by the second radar. 7-5. A program according to any one of 7-1. to 7-4., wherein the second image is an image obtained by cutting out a part of an image generated using the second radar signal data. 7-6. A program according to 7-5., wherein the second image is an image obtained by cutting out a region corresponding to the first image from an image generated using the second radar signal data. 7-7. A program according to any one of 7-1. to 7-6., wherein the second image has a higher resolution than the first image. 7-8. A program according to any one of 7-1. to 7-7., wherein the learning means generates a model by performing machine learning using a plurality of the first images, which are a plurality of video frames, and a plurality of the second images, which are also a plurality of video frames. 8-1. 8-2. A program causing a computer to function as learning means for generating a model by performing machine learning using a first image and a second image, wherein the first image is an image generated using first radar signal data obtained by a first radar, and the second image is an image generated using second radar signal data obtained by a second radar, and the center frequency of the frequency band of radio waves transmitted and received by the second radar is higher than the center frequency of the frequency band of radio waves transmitted and received by the first radar. 8-2. A program in accordance with 8-1., wherein the aperture size of the second radar is the same as the aperture size of the first radar. 8-3. A program in accordance with 8-1. or 8-2., wherein the second image has a higher resolution than the first image. 8-4. A program in accordance with any one of 8-1. to 8-3., wherein the learning means generates a model by performing machine learning using a plurality of the first images, which are a plurality of video frames, and a plurality of the second images, which are also a plurality of video frames.9-1. A program causing a computer to function as conversion means for converting a target image using a model generated by the model generation device described in any one of 1-1 to 1-8. 9-2. A program causing a computer to function as conversion means for converting a target image using a model generated by the model generation device described in any one of 1-2 to 1-4. 9-3. A program causing a computer to function as conversion means for converting a target image using a model generated by the model generation device described in any one of 2-1 to 2-4. 9-4. A program in the program described in 9-2 or 9-3, wherein the target image is an image obtained by a radar having the same configuration as the first radar. 10. A computer-readable recording medium having recorded thereon a program described in any one of 7-1 to 7-8. 11. A computer-readable recording medium having recorded thereon a program described in any one of 8-1 to 8-4. 12. A computer-readable recording medium having recorded thereon a program described in any one of 9-1 to 9-4.

[0120] This application claims priority based on Japanese Patent Application No. 2023-095343, filed on June 9, 2023, the disclosure of which is incorporated herein by reference in its entirety.

[0121] 10 Model generation device 20 Conversion device 30 Model 50 Subset 51 First radar 52 Second radar 61 First image 62 Second image 81 First memory unit 82 Model memory unit 83 Second memory unit 84 Output data memory unit 110 Acquisition unit 120 Training data generation unit 130 Learning unit 210 Target data acquisition unit 230 Conversion unit 501 Transmitting antenna 502 Receiving antenna 503 Transmitting and receiving antennas 510, 520 Aperture 1000 Computer

Claims

1. a learning means for generating a model by performing machine learning using the first image and the second image; the first image is an image generated using first radar signal data; the second image is an image generated using second radar signal data; The first radar signal data is a signal extracted from the second radar signal data. Model generation device.

2. 2. The model generating device according to claim 1, the second radar signal data is signal data obtained by a second radar including the first radar, The first radar signal data is signal data obtained by extracting data corresponding to the signal data of the first radar from the second radar signal data. Model generation device.

3. 3. The model generating device according to claim 1, The second image has a higher resolution than the first image. Model generation device.

4. 3. The model generating device according to claim 1, The learning means generates a model by performing machine learning using the first images, which are a plurality of video frames, and the second images, which are a plurality of video frames. Model generation device.

5. The model generating device according to claim 1 or 2 further comprises a conversion means for converting a target image using the model generated by the model generating device according to claim 1 or 2. Conversion device.

6. one or more computers perform machine learning using the first image and the second image to generate a model; the first image is an image generated using first radar signal data; the second image is an image generated using second radar signal data; The first radar signal data is a signal extracted from the second radar signal data. Model generation method.

7. 7. The model generation method according to claim 6, The second image has a higher resolution than the first image. Model generation method.

8. 7. The model generation method according to claim 6, The one or more computers generate the model by performing machine learning using the first images, which are frames of a video, and the second images, which are frames of a video. Model generation method.

9. One or more computers convert the target image using the model generated by the model generation device according to claim 1 or 2. How to convert.

10. A computer is caused to function as a learning means for generating a model by performing machine learning using a first image and a second image; the first image is an image generated using first radar signal data; the second image is an image generated using second radar signal data; The first radar signal data is a signal extracted from the second radar signal data. program.