Object detector and object detection method

Abstract

To improve the accuracy of detection of an object.SOLUTION: An object detector 100 detects an object. The object detector 100 comprises: an observation data acquisition section 101; a related data acquisition section 102; an information shaping section 103; an inference section 107; and a display section 106. The observation data acquisition section acquires observation data obtained by a main sensor 110 upon detection of an object. The related data acquisition section acquires related data containing parameters of the main sensor and environment data indicating an observation environment. The information shaping section generates shaping data obtained by expanding the related data or a result of processing the related data using at least a part of a discriminator, to a shape capable of being inputted to the discriminator. The inference section infers an object according to convolution processing by inputting the related and the shaping data to the discriminator. The display section displays an inference result according to the inference section.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an object detection device and an object detection method. [Background technology]

[0002] In order to search for target objects over a wide area and use their location information to recover or avoid them, research is being conducted into attaching sensors to moving objects such as ships, automobiles, and unmanned aircraft, and detecting target objects using data obtained from the sensors.

[0003] There is a demand for highly accurate detection of target objects while moving over a wide area or in various locations under different conditions, and in recent years, machine learning techniques have been applied.

[0004] However, the observation data used to search for target objects varies greatly in nature depending on factors such as the amount of light, the position of the light source, topography, geology, temperature, and noise, so a single classifier may not be able to detect the target object with high accuracy.

[0005] Non-Patent Document 1 describes inference using additional data in image and numerical formats in addition to the images of observed data. In this document, image format data is vectorized using ResNet50, and numerical format data is collected as is and aligned in a row, and each data is added to the observed data by full connection. [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] Yusuke Nishizawa, Shinichi Homma, Hayato Tobe, Yasuyuki Miyajima, Daisuke Fukushima, Prediction of Tunnel Face Spalling Using Multimodal Deep Learning, 2019 Annual Conference of the Japanese Society for Artificial Intelligence, p.2 Summary of the Invention [Problem to be solved by the invention]

[0007] In order to achieve high accuracy in object detection across a variety of situations with a single classifier, it is possible to use related data, including data indicating the observation environment (hereafter referred to as environmental data) and sensor parameters, in addition to the observation data. By using related data, information about the observation environment that cannot be obtained from the observation data can be obtained, allowing the classifier to detect objects in accordance with the observation environment, i.e., the characteristics of the observation data.

[0008] However, there are issues to consider when utilizing related data. One of the issues is when the formats and properties of the observed data and the related data are different. In object detection, the observed data is generally image data that contains location information. In contrast, related data can be scalar values ​​such as the exposure time of a camera or the speed of a moving object, which do not contain location information and relate to the entire observed data. Therefore, it is desirable to be able to apply related data uniformly to the observed data.

[0009] Another issue is the location where related data should be applied. In recent years, deep learning has become the mainstream method used for object detection, and since processing is divided into layers and branching and integration are possible, it is desirable to determine the location (layer) where related data should be added (integrated) for each piece of data.

[0010] However, in Non-Patent Document 1, related data is added to observed data by full coupling at one time, and therefore the above demands cannot be met.

[0011] The present invention has been made in view of the above-mentioned problems, and has an object to provide a technique for improving the accuracy of object detection. [Means for solving the problem]

[0012] In order to achieve the above object, the present invention provides an object detection device for detecting an object, comprising: an observation data acquisition unit for acquiring observation data of the object detected by a main sensor; an associated data acquisition unit for acquiring associated data including parameters of the main sensor and environmental data indicating the observation environment; a shaping unit for generating shaped data by expanding the associated data or the results of processing the associated data by at least a part of an identifier into a form that can be input to the identifier; an inference unit for inputting the associated data and the shaped data to the identifier and inferring the object by convolution processing; and a display unit for displaying the inference results by the inference unit. [Effects of the Invention]

[0013] According to the present invention, it is possible to improve the accuracy of object detection. [Brief explanation of the drawings]

[0014] [Figure 1] FIG. 1 is a schematic block diagram illustrating an example of the configuration of an object detection device according to a first embodiment. [Figure 2] FIG. 1 is a block diagram illustrating an example of a hardware configuration of an object detection device according to a first embodiment. [Figure 3] FIG. 1 is a diagram showing an example of the configuration of an onboard object detection device according to a first embodiment. [Figure 4] 10 is a flowchart illustrating an example of a learning process according to the first embodiment. [Figure 5] 10 is a flowchart illustrating an example of an environmental information addition process according to the first embodiment. [Figure 6] 10 is a flowchart showing another example of the environmental information addition process according to the first embodiment. [Figure 7] 10 is a flowchart illustrating an example of an object detection process according to the first embodiment. [Figure 8] FIG. 3 is a diagram showing an example of a display screen of the output device according to the first embodiment. [Figure 9] 10 is a flowchart illustrating an example of a learning process according to the second embodiment. [Figure 10] 10 is a flowchart illustrating an example of an object detection process according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0015] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The following description and drawings are examples for explaining the present invention, and some omissions and simplifications have been made as appropriate for clarity of explanation. The present invention can be implemented in various other forms. Unless otherwise specified, each component may be singular or plural.

[0016] The object detection device of the present invention can be applied to, for example, a fish finder that uses sound waves. In the following, a fish finder will be described as an example of the object detection device. However, the object detection device of the present invention is not limited to a fish finder.

[0017] An underwater object detection device such as a fish finder detects objects (fish in the case of a fish finder) by emitting sound waves underwater, observing the reflected waves, and converting the observed data into image data for display, or by inputting the observed data into a classifier. Underwater object detection devices are used, for example, in rivers, oceans, lakes, ponds, and other underwater locations, as well as in places where water has accumulated in artificial structures such as fish pens or pools.

[0018] The related data includes information about the measurement environment (water depth, water temperature, bottom sediment, air temperature, wind speed, wind direction, salinity concentration, etc.) and the settings of the main sensor (observation direction, gain, etc.). In other words, the related data may be detected by the main sensor or sub-sensor, or may be obtained from a database or by manual observation.

[0019] Furthermore, the object detection device can be applied not only underwater but also in spaces or materials where sound, light, radio waves, etc. propagate, such as on land, in the air, underground, etc. Note that, as long as the observation data can be converted into an image, it is not limited to observation data obtained by observing with sound waves, but can also be applied to observation data obtained by a wide range of search methods, such as radio waves, taking images with a camera, magnetic exploration, or electromagnetic induction exploration.

[0020] Furthermore, the sensors that collect observation data and the sensors that acquire related data are not limited to being mounted directly on moving equipment such as a ship, but may be mounted on objects that move synchronously or asynchronously with the moving equipment. Furthermore, these sensors and sub-sensors can also be applied to cases where the sensors are fixed at fixed positions in a river, such as for detecting fish in a river. [Example]

[0021] The first embodiment is an example in which a target object present in water is detected while the ship is moving or stopped.

[0022] FIG. 1 is a schematic block diagram illustrating an example of the configuration of an object detection device according to a first embodiment.

[0023] The object detection device 100 can be realized by a high-performance calculator, personal computer, or mobile terminal. The object detection device 100 includes an observation data acquisition unit 101, an associated data acquisition unit 102, an information shaping unit 103 (an example of a "shaping unit"), an input interface 104, an output interface 105, a display unit 106, a learning unit 107, and an inference unit 108. The observation data acquisition unit 101, the associated data acquisition unit 102, the information shaping unit 103, the input interface 104, the output interface 105, the display unit 106, the learning unit 107, and the inference unit 108 are communicatively connected via a bus 109. Furthermore, the object detection device 100 is communicatively connected to a main sensor 110, a sub-sensor 111, a database 112, an input device 113, and an output device 114 via wired or wireless connections.

[0024] The main sensor 110 is installed on the ship 310 (see FIG. 3). The main sensor 110 emits sound waves into the water and detects the reflected sound waves as observation data. The data acquisition unit 101 performs preprocessing such as signal normalization on the observation data detected by the main sensor 110 and converts the data into images. Note that the main sensor 110 may also process the observation data up to imaging. Furthermore, the main sensor 110 may have a function to acquire environmental data indicating the observation environment, such as the distance from the water surface or bottom and water temperature, at the location where the sound wave emitting part is set.

[0025] The sub-sensor 111 may measure a single value such as temperature, or may measure various values ​​in addition to temperature such as humidity, wind direction and wind speed, or may measure image-like data such as water temperature and current measured for each water depth and direction based on the ship 310, or may be a collection of such sensors.

[0026] The database 112 stores environmental data relating to the observation environment that has been measured in advance. For example, nautical charts, i.e., water depth and bottom sediment data, are stored in the database 112. The water depth and bottom sediment data in the observation environment can be acquired based on the position information of the ship 310.

[0027] The input device 113 may be a keyboard, a touch panel, a card reader, or a voice input device. The input device 113 accepts input from a user. Items input to the input device 113 include parameters to be set in the classifier as well as data observed by a person. These parameters to be set in the classifier and the data observed by a person can be input to the input device 113 as related data. The input interface 104 transmits these parameters to be set in the classifier and the data observed by a person to each section.

[0028] The related data acquisition unit 102 acquires related data from the main sensor 110, the sub-sensor 111, the database 112, and the input device 113. The source of the related data may be all four of the main sensor 110, the sub-sensor 111, the database 112, and the input device 113, or one or more of the four may be selected. The related data may include parameters of the main sensor and environmental data indicating the observation environment.

[0029] The information shaping unit 103 performs preprocessing such as normalization on the related data acquired by the related data acquisition unit 102, and arranges the data in a form that can be input to a classifier. That is, the information shaping unit 103 expands the related data into a form that can be input to a classifier, and generates shaped data. The shape, size, and number of the shaped data depend on the classifier. The information shaping unit 103 may generate shaped data by expanding the result of processing the related data with at least a part of the classifier into a form that can be input to the classifier.

[0030] The learning unit 107 uses a machine learning technique to learn a classifier that determines whether or not a target object, which is an example of a "target object," is captured in the observation data, based on the observation data obtained via the observation data acquisition unit 101 and the shaped data obtained by the information shaping unit 103 (learning process). At this time, the learning unit 107 may specify the configuration of the classifier, the associated data to be used, and the position to add the associated data via the input interface 104, or may use a classifier with a default configuration if these are not specified.

[0031] Similar to learning unit 107, inference unit 108 determines whether a target object is captured in the observation data based on the observation data obtained via observation data acquisition unit 101 and the shaped data obtained by information shaping unit 103. Inference unit 108 may output only whether or not a target object has been detected, or may output additional information such as the position, size, or score representing the reliability of detection of the detected target object.

[0032] The output device 114 may be a display, a speaker, a printer, or the like. The output device 114 outputs data acquired via the output interface 105 to a user. The output interface 105 is communicably connected to the output device 114 and is capable of outputting characters, symbols, images, videos, and the like to the output device 114. The output device 114 displays various information related to object detection, such as an image of the observation data to be analyzed and the coordinates of detected objects. The output device 114 may display the coordinates of the detected object as is, or may display a mark indicating the position of the object superimposed on the image of the observation data.

[0033] The output device 114 is not limited to a device directly connected to the object detection device 100, but may be a device capable of displaying a screen, such as a personal computer, tablet, or smartphone used by a user. In this case, the screen information may be transmitted from the output interface 105 to the device having a display screen by wireless communication or via a network.

[0034] Thus, object detection device 100 is not limited to being configured as a standalone computer and peripheral devices connected to it via a wire. Object detection device 100 may also be configured as a system in which it is combined with an external information device (such as the above-mentioned tablet, smartphone, or another computer) that can be connected to object detection device 100 via a wireless or wired network.

[0035] Fig. 2 is a block diagram illustrating an example of a hardware configuration of an object detection device according to Example 1. The same components as those in Fig. 1 are denoted by the same reference numerals, and the description thereof will be omitted.

[0036] 1 includes a CPU (Central Processing Unit) 201, a memory 202, a storage device 203, a communication unit 204, an input interface 104, and an output interface 105. The CPU 201, the memory 202, the storage device 203, the communication unit 204, the input interface 104, and the output interface 105 are connected via a bus 109 so as to be able to communicate with each other.

[0037] CPU 201 is a central processing unit that implements necessary functions by executing programs stored in memory 202 (or storage device 203). Memory 202 is a main storage device used by CPU 201 when executing processing, and may be a volatile storage element such as RAM (Random Access Memory). Storage device 203 is an auxiliary storage device for storing input data provided to CPU 201 and output data output from CPU 201, and is configured with a non-volatile storage element such as an HDD (Hard Disk Drive) or SSD (Solid State Drive).

[0038] Communication unit 204 is an interface used by object detection device 100 to communicate with external devices, and may be a network interface card (NIC), etc. Communication unit 204 is connected to a network (e.g., the Internet (registered trademark)) and communicates with external devices via the network.

[0039] The functions of the observation data acquisition unit 101, the related data acquisition unit 102, the information shaping unit 103, the learning unit 107, and the inference unit 108 can be realized by having the CPU 201 execute specific programs. Furthermore, instead of a CPU, they may be realized by a GPU (Graphics Processing Unit), a dedicated hardware circuit, or a combination of a hardware circuit and software. The bus 109 mediates the transmission of data handled by each processing unit connected to the bus 109, as well as control data and analysis data.

[0040] FIG. 3 is a diagram illustrating an example of the configuration of the object detection device according to the first embodiment.

[0041] The object detection device 100 may be installed on a ship 310. A main sensor 110 and a sub-sensor 111 are installed on the ship 310. Note that the object detection device 100, database 112, input device 113, and output device 114 are not explicitly shown in FIG. 3 . The object detection device 100, database 112, input device 113, and output device 114 may be installed inside the ship 310. The main sensor 110 performs observation using sonar signals and detects an object 311.

[0042] FIG. 4 is a flowchart illustrating an example of the learning process according to the first embodiment.

[0043] The learning process is a process for learning a classifier used in the object detection process in object detection device 100. The learning process can be realized by CPU 201 executing a computer program stored in memory 202 (or storage device 203). The learning process is roughly divided into a data acquisition phase from steps S401 to S406 and a learning phase from steps S410 to S416.

[0044] First, the user sets up the main sensor 110 and the sub-sensor 111 via the input interface 104 and determines the conditions for acquiring the observation data and related data to be acquired (step S401). The settings include the sensitivity of the main sensor 110, the acquisition interval, and the selection of related data to be saved. The settings are saved in the storage device 203.

[0045] Next, the observation data acquisition unit 101 and the related data acquisition unit 102 acquire various data (steps S402 to S406).

[0046] The observation data acquisition unit 101 receives observation data from the main sensor 110 (step S403), and the related data acquisition unit 102 receives related data from the main sensor 110, the sub-sensor 111, and the input device 113 (step S404). The observation data is sound wave data observed by the main sensor 110. The related data is data obtained from the main sensor 110, the sub-sensor 111, or the input device 113, such as the distance and water depth of the main sensor 110 from the water surface, temperature, humidity, wind direction, and wind speed.

[0047] Here, the observation data and environmental data are acquired at predetermined time intervals (for example, every few seconds). For example, the observation data acquisition unit 101 acquires observation data for a certain point at regular or irregular time intervals by sequentially observing the entire observation area while moving in a specific direction.

[0048] The observation data acquisition unit 101 and the related data acquisition unit 102 record the received data in the storage device 203 (step S405). The data acquired from the main sensor 110, the sub-sensor 111, and the input device 113 is recorded together with the time or a value equivalent thereto, such as a count value that is incremented each time observation data is acquired, and the observation data is associated with the related data.

[0049] If the observation data acquisition unit 101 and the related data acquisition unit 102 are unable to acquire related data due to a problem with the sensors, etc., they may store a value equivalent to the missing value based on conditions such as a time threshold set for each piece of environmental information.

[0050] When the conditions are met, the observation data acquisition unit 101 and the related data acquisition unit 102 terminate data acquisition (step S406). The termination conditions may be the completion of a data acquisition plan formulated in advance, the passage of time, the ship leaving a specific area, or the number of acquired data exceeding a threshold value.

[0051] Furthermore, information on the location of the target object based on the observation data may be created and saved at the same time as the data is acquired, or may be created and saved after the data acquisition is complete.

[0052] The learning phase will now be described. First, the user sets learning settings (step S410). The settings include settings related to the architecture, such as the layer structure of the classifier or the related data to be used, and the method and location of adding the related data, as well as settings related to learning parameters, such as the number of learning iterations or the loss function.

[0053] The architecture settings can be created and input each time, or can be selected from templates created in advance. Designing an architecture requires knowledge of the characteristics of the main sensor 110 and the machine learning method used, making it difficult for non-experts to do. By selecting from templates, even non-experts can create a classifier with high accuracy.

[0054] The learning unit 107 and the information reforming unit 103 perform learning of the classifier (steps S411 to S415).

[0055] First, the learning unit 107 performs learning start processing such as construction and initialization of the classifier architecture (step S411).

[0056] Next, the learning unit 107 reads data and performs preprocessing (step S412). As the related data, only the data set in step S410 is read from the data recorded in the storage device 203 in step S405 or the data recorded in the storage device 203 as the database 112. Preprocessing is performed on both the observed data and the related data, and processing is performed so that the ranges of many values ​​fall within a certain range.

[0057] The information formatting unit 103 formats the related data to match the shape of the observation data (step S413).

[0058] When dealing with image-like data in deep learning, convolution processing is commonly used, as it can preserve spatial relationships within the image. However, related data can include scalar values ​​such as temperature, water depth, and gain. In multimodal methods that combine multiple data sets, including scalar values, full connection processing is used, which results in the loss of spatial relationships preserved by convolution processing. Furthermore, full connection updates only the nodes activated by the training data, making it impossible to consider data related to the entire observation data, such as gain, as a whole.

[0059] By using the method shown in Figure 5 as an example, the related data can be expanded and shaped to correspond to the observed data, thereby maintaining the positional relationship of the observed data and uniformly adding the related data to the observed data.

[0060] FIG. 5 is a flowchart illustrating an example of an environmental information addition process according to the first embodiment.

[0061] This example shows the process of adding environmental information, in which related data is added midway through the classifier. Observation data 501 is input to the first half 502 of the classifier. The first half 502 of the classifier is a neural network including a convolutional layer. The first half 502 of the classifier obtains intermediate output 503 with vertical x, horizontal y, and channel h.

[0062] Meanwhile, environmental information 510 is input to a fully connected layer 511. In the figure, each circle represents one piece of related data. The fully connected layer 511 obtains a vectorized signal 512 with a size of 1 vertically, 1 horizontally, and channel h'. At this time, by adding the value of any related data itself or a normalized version of that value in the channel direction (channel size h'+1), it is possible to treat specific related data with emphasis. By expanding and combining the vectorized signal 512 by x x y pieces, a shaped signal 513 with a size of x vertically, y horizontally, and channel h' is obtained. This shaping process causes the intermediate output 503 and the shaped signal 513 to match in the size of x vertically and y horizontally.

[0063] The intermediate output 503 and the shaped signal 513 are combined in the channel direction to obtain a vertical x, horizontal y, channel (h+h') signal 520. This signal 520 is input to the latter half 521 as part of the "classifier" of the classifier, and object detection is performed.

[0064] Since the convolution process performs the same processing for all vertical x and horizontal y dimensions, this shaping process applies the same vectorized signal 512 to each of the vertical x and horizontal y dimensions of the intermediate output 503, allowing related data to be added uniformly. Furthermore, since the weights to be learned only need to be the size of the convolution window, memory usage can be reduced compared to fully connected processing in which x×y×h+h' is converted to x×y×(h+h'). Furthermore, compared to when related data is not used, the weights only increase in the channel direction, so parameters such as the window size in the convolution process can be used as is.

[0065] Furthermore, since the location information of the observation data can be maintained, the location where related data is added can be freely determined.

[0066] FIG. 6 is a flowchart illustrating another example of the environmental information addition process according to the first embodiment.

[0067] This example shows the process of adding environmental information, in which related data is added at the input stage, before the observed data is input to the classifier. The processing of the related data (up to obtaining the shaped signal 513) is the same as in Figure 5. However, the vertical x and horizontal y dimensions are the same as the size of the observed data. The observed data (channel size C) and the shaped signal 513 are combined to obtain a signal 530 with vertical x, horizontal y, and channel (C+h'). This signal 430 is input to the classifier 431, and object detection is performed.

[0068] As shown in Figures 5 and 6, related data can be added at any layer, eliminating the need to add environmental information all at once. Instead, each piece of related data can be added at the point where it is most effective in improving the target object detection rate, allowing for the construction of a more accurate classifier.

[0069] The position where the related data is added can be determined by taking into consideration the characteristics of the related data, such as "the gain of the gain amplifier is directly related to the signal strength of the observed data, so it is added at the input stage," or it can be selected randomly or from patterns based on the results of prior verification.

[0070] Returning to step S414 in Fig. 4, the learning unit 107 learns the classifier of the above-described architecture using observed data and related data.

[0071] The learning unit 107 ends learning of the classifier when a learning end condition is met (step S415). The learning end condition may be when the number of learning rounds reaches a predetermined number, or when the accuracy rate in the validation data exceeds a threshold.

[0072] The learning unit 107 records the classifier for which learning has been completed in the storage device 203 (step S416). At this time, it also records what architecture and what associated data has been added to which layer.

[0073] FIG. 7 is a flowchart illustrating an example of an object detection process according to the first embodiment.

[0074] The inference unit 108 executes the object detection process using the trained classifier.

[0075] First, the inference unit 108 reads and applies settings for inference (step S701). Information about the classifier to be used (such as the architecture, the associated data added to which layer, and the weight parameters of the trained classifier) ​​is read and applied from the storage device 203 and the input interface 104. The inference unit 108 may also read and apply criteria for which of the classifier outputs to adopt, such as a score threshold representing the reliability of detection.

[0076] The inference unit 108 starts inference based on the setting in S701 (step S702).

[0077] The inference unit 108 receives the observation data and related data and performs preprocessing (step S703). The inference unit 108 receives the observation data from the main sensor 110, and also receives related data from the main sensor 110, the sub-sensor 111, the database 112, and the input device 113. At this time, the related data to be received is limited to the data specified in step S701.

[0078] The inference unit 108 shapes the pre-processed received signal, which is the same as step S413.

[0079] The inference unit 108 performs inference based on the observation data and related data (step S704). The classifier may perform object detection using the observation data and related data as input. The display unit 106 may output not only whether or not a target object has been detected, but also, if detected, its location and size (step S705).

[0080] Finally, inference unit 108 determines whether to end the process based on a preset condition for ending the process (step S706). The end condition may be that object detection device 100 detects the target object and it is visible to a human, that the target object is avoided, that the target object is collected and no longer exists, that main sensor 110 has finished observation, or that the user has performed an end operation. If it is determined in step S706 that the process should be ended, inference unit 108 ends the object detection process.

[0081] FIG. 8 is a diagram illustrating an example of a display screen of the output device according to the first embodiment.

[0082] A display screen 800 is displayed on the output device 114. The display screen 800 simultaneously displays the content displayed by the input interface 104 for setting the conditions for the processes executed by the learning unit 107 and the inference unit 108, and the content displayed by the output interface 105 for outputting the inference results of the inference unit 108.

[0083] First, input fields 801 and 802 corresponding to the setting screen of the input interface 104 will be described.

[0084] The input field 801 is a parameter field for setting parameters used by the classifier when it performs learning or inference operations. The parameters to be set include the related data to be used, the location to add the related data, normalization of observed data for gain adjustment, specification of the range of observed data to be input to the classifier, and the maximum number of detections. In the input field 801, any of the multiple parameters may be changed, or the default parameters may be used. In this example, information Z is set to 3. When related data is entered manually, or the update interval may differ for each related data, the related data item may have a function to display how long ago the value was entered.

[0085] Input field 802 is a field for setting a classifier. Input field 802 displays candidate architectures of classifiers that object detection device 100 learns, or classifiers stored in storage device 203 for use in inference. In this example, classifier 1 is set as the classifier. Furthermore, during inference, the input information to be used in input field 801 is determined based on the selection of the classifier here.

[0086] Next, an output field 810 corresponding to the result display by the output interface 105, an observed image 811, and a detection result 812 as an example of an "inference result" will be described.

[0087] The display field 810 is a field for displaying the value of the received associated data 510 and status such as the detection situation. In addition to manual input, the associated data also includes data obtained from the main sensor 110, sub-sensor 111, or database 112. Therefore, displaying this data can serve as a reference for changing the classifier used for inference. In this example, the values ​​of information A and information B are displayed, but information C is not displayed because it has not been obtained.

[0088] The observed image 811 is an image of the observed data, and when the main sensor 110 is moving while making observations, the image is scrolled and displayed each time the observed data is read in. In this example, three detection results 812 by the classifier are displayed superimposed on the observed image 811.

[0089] The maximum detection number set in the input field 801 represents the maximum number of detection results 812 that will be displayed on the display screen 800. In this example, the maximum detection number is set to 3, and three detection results 812 are displayed. If the classifier cannot detect the target object, the detection results 812 are not displayed. Note that if the detection results are transmitted to the control of the ship or robot and the ship or robot operates based on the results, the display field 810, observed image 811, and detection results 812 do not necessarily have to be displayed.

[0090] The method of selecting the detection result to be displayed may be to simply use the one with the highest score representing the reliability of the detection, or may use a condition such as the existence of a nearby detection solution with a high score in the previous time.

[0091] As described above, according to this embodiment, in addition to the observation data, information about the measurement environment, the settings of the main sensor, and the like are input as related data to the classifier and inference is performed, thereby making it possible to improve the accuracy of object detection when detecting objects from the observation data. [Example]

[0092] In the first embodiment, inference was performed using a predetermined classifier in step S704. As described above, when a sensor is installed on a mobile body and observations are made while moving, the situation may change significantly. For example, the bottom sediment may change from sand to mud, the bottom undulations may become gentler, the water temperature may drop, and the salinity concentration may change, which may change the way sound waves travel or are reflected, significantly affecting the appearance of the observation data. Because related data is acquired, it is possible to infer the appearance of the observation data.

[0093] Furthermore, increasing the versatility of a classifier to accommodate many situations can sometimes result in a trade-off with accuracy. Therefore, when highly accurate object detection is desired, multiple classifiers may be prepared rather than consolidating them into a single classifier. However, if an appropriate classifier is used, target objects can be detected with high accuracy. However, if an inappropriate classifier is used, it may not be possible to detect the target object, or background noise unrelated to the target object may be mistakenly detected as the target object. Therefore, it is important to select an appropriate classifier depending on the situation.

[0094] In the second embodiment, an example of a method for selecting a classifier during object detection will be described.

[0095] FIG. 9 is a flowchart illustrating an example of a learning process according to the second embodiment.

[0096] The hardware configuration of object detection device 100 that executes the processes shown in Fig. 9 and Fig. 10 described later is the same as the configuration shown in Fig. 2, and can be realized by modifying the program that causes Example 1 to function. Furthermore, the processes shown in Fig. 9 and Fig. 10 can be realized by CPU 201 of object detection device 100 executing a computer program stored in memory 202 (or storage device 203). Note that the same steps as those in Fig. 4 and Fig. 7 are denoted by the same reference numerals, and descriptions thereof will be omitted.

[0097] The process up to data acquisition (step S406) is the same as in the first embodiment, and therefore a description thereof will be omitted. After data acquisition is completed, the learning unit 107 performs learning of a plurality of classifiers. The main part of the learning phase (steps S410 to S416) is the same as in the first embodiment. The difference from the first embodiment is steps S901 and S902, in which the main part of the learning phase (steps S410 to S416) is repeated multiple times.

[0098] The number of classifiers to be trained and the parameters to be used for training may be determined via the input interface 104, or may be determined dynamically based on an index such as the coverage rate of correct cases for the validation data.

[0099] When the classifier is recorded in step S416, information such as the range of the observed data and related data used in the learning is also recorded. These are used to select a classifier, as will be described later.

[0100] FIG. 10 is a flowchart illustrating an example of an object detection process according to the second embodiment.

[0101] 7 is that a step of selecting a classifier (step S1001) has been added. After acquiring the observed data and related data (step S703), the inference unit 108 selects a classifier.

[0102] In selecting a classifier, the inference unit 108 compares the observed data and related data acquired in step S703 with the observed data and related data for each classifier recorded in step S416 and determines the classifier to be used. The inference unit 108 may use a classifier whose observed data and related data fall within a range. Furthermore, when multiple classifiers fall within the range, the inference unit 108 may preferentially use the currently used classifier, or may determine the classifier to be used based on the most recent number of times it has been used or the detection rate.

[0103] 10, a classifier is selected each time observation data and related data are acquired. However, instead of selecting a classifier every time, a classifier may be selected at regular time intervals or in response to a manual operation via the input interface 104.

[0104] Although the present invention has been described above based on the embodiments, the present invention is not limited to the above-described embodiments and various modifications are possible within the scope of the present invention. For example, the observation data to be visualized is not limited to the observed values ​​of sound waves, but may be image data acquired from a camera or the like, or a radar image manipulated by radio waves. In addition to images, time-series data that quantifies the state of the observation target may also be applied. [Explanation of symbols]

[0105] 100...object detection device, 101...observation data acquisition unit, 102...related data acquisition unit, 103...information shaping unit, 106...display unit, 107...learning unit, 108...inference unit, 110...main sensor, 211...main sensor, 311...object, 501...observation data, 502...first half, 510...related data, 521...second half, 531...classifier, 812...detection result

Claims

1. An object detection device for detecting an object, an observation data acquisition unit that acquires observation data of the object detected by the main sensor; a related data acquisition unit that acquires related data including parameters of the main sensor and environmental data indicating an observation environment; a shaping unit that generates shaped data by expanding the associated data or a result of processing the associated data by at least a part of a classifier into a form that can be input to the classifier; an inference unit that inputs the related data and the shaped data to the classifier and infers the object by convolution processing; and a display unit that displays an inference result by the inference unit.

2. a learning unit that learns the observation data and the shaped data, The object detection device according to claim 1 .

3. the observation data acquisition unit acquires observation data of the object in the water detected by the main sensor; The object detection device according to claim 1 .

4. the inference unit replaces a part of the related data with a value that probabilistically represents a defect; The object detection device according to claim 1 .

5. the learning unit determines an input position to the classifier for each of the plurality of related data. The object detection device according to claim 2 .

6. the learning unit learns a plurality of patterns in which whether or not input to the classifier and an input position are set for each of the plurality of pieces of related data; The object detection device according to claim 2 .

7. the inference unit selects a classifier trained by the learning unit from among the plurality of classifiers. The object detection device according to claim 2 .

8. the display unit displays the associated data input to the inference unit. The object detection device according to claim 1 .

9. the display unit displays the detection result of the object inferred by the inference unit. The object detection device according to claim 1 .

10. An object detection method using an object detection device that detects an object, acquiring observation data of the object detected by the main sensor; acquiring parameters of the main sensor and related data including environmental data indicative of an observation environment; generating shaped data by expanding the associated data or a result of processing the associated data by at least a part of a classifier into a form that can be input to the classifier; learning the observed data and the shaped data; inputting the associated data and the shaped data into the classifier and inferring the object by convolution processing; and displaying the results of the inference. ?

Images