Image recognition device, system and method, and conversion data setting method

Abstract

We aim to improve the accuracy of object detection by using multiple image recognition models. [Solution] First and second object detection data are generated by performing object detection processing using first and second image recognition models, and inference result data is generated based on these. The first object detection data, second object detection data, and inference result data each include class data indicating the type of object in the object detection area and position and shape data of the object detection area. If the class data in the second object detection data indicates a specific type, the class data in the second object detection data is converted to other class data according to a pre-set conversion rule. When the class data in the first object detection data matches other class data, position and shape data in the inference result data is generated by integrating the position and shape data in the first and second object detection data.

Description

[Technical Field]

[0001] The present invention relates to an image recognition device, system, and method, as well as a method for setting converted data. [Background technology]

[0002] A technique is known that uses machine learning models to perform data classification processing through the derivation of feature vectors (see, for example, Patent Document 1 below). Furthermore, image recognition models capable of performing object detection can identify object detection regions in addition to detecting the class of objects in an image. These object detection regions are often referred to as bounding boxes. [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2023-27858 [Overview of the project] [Problems that the invention aims to solve]

[0004] One approach to improving the accuracy of object detection is to prepare multiple image recognition models capable of performing object detection processing and integrate the data (position and shape data) of each object detection region identified by these multiple image recognition models. However, this integration presupposes the matching of multiple class data corresponding to multiple object detection regions. However, when multiple existing image recognition models are prepared, the derived class data (class names) often differ among the multiple image recognition models, even if the objects to be detected are the same. As a result, the improvement in accuracy of object detection through integration is often hindered.

[0005] The present invention aims to improve the effectiveness of increasing the accuracy of object detection using multiple image recognition models. [Means for solving the problem]

[0006] The image recognition device according to the present invention has a controller that performs inference for recognizing an object to be recognized in an image. The controller has a first image recognition model and a second image recognition model, and by performing object detection processing using the first image recognition model and the second image recognition model respectively to detect an area in the input image where the object to be recognized is determined to exist as an object detection area, the controller generates first object detection data with the first image recognition model and second object detection data with the second image recognition model, and generates inference result data based on at least one of the first object detection data and the second object detection data. The first object detection data, the second object detection data and the inference result data each include class data indicating the type of object in the object detection area and position and shape data of the object detection area. The controller, when the class data in the second object detection data indicates a specific type, converts the class data in the second object detection data to other class data according to a preset conversion rule, and when the class data in the first object detection data matches the other class data, sets the class data in the first object detection data or the other class data as the class data in the inference result data, and generates the position shape data in the inference result data by integrating the position shape data in the first object detection data and the position shape data in the second object detection data. [Effects of the Invention]

[0007] In inference, object detection processing is performed by each image recognition model, and by integrating the position and shape data in each object detection data, an object detection region that accurately represents the position and size of the object to be recognized in the input image can be derived. However, when configuring an image recognition device by combining multiple existing image recognition models, different class data is often derived from each image recognition model for a common object to be recognized. Taking this into consideration, the image recognition device according to the present invention enables the conversion of class data according to pre-set conversion rules. In cases where different class data is derived from each image recognition model for a common object to be recognized, conversion rules can be defined so that the class data matches among the multiple image recognition models through class data conversion. As a result, even when different class data is derived from each image recognition model for a common object to be recognized, it becomes possible to derive a highly accurate object detection region through the integration of position and shape data after class data matching. [Brief explanation of the drawing]

[0008] [Figure 1] This diagram illustrates an embodiment of the present invention, showing the relationship between the vehicle and other components. [Figure 2] This is an internal block diagram of a driver assistance device according to an embodiment of the present invention. [Figure 3] The present invention relates to an embodiment of the present invention, and shows (a) and (b) functional block diagrams of an in-vehicle controller focusing on image recognition, and (c) and (d) configuration diagrams of object detection data and inference result data generated by the in-vehicle controller. [Figure 4] This figure illustrates an embodiment of the present invention, showing the position and shape of an object detection region in a two-dimensional image. [Figure 5] This figure shows how an object detection region is set for a target object to be recognized in a two-dimensional image, according to an embodiment of the present invention. [Figure 6] This is a schematic diagram of the first reference case related to the integration of object detection regions. [Figure 7] This is a schematic diagram of the second reference case related to the integration of object detection regions. [Figure 8] This is an internal block diagram of a server device according to an embodiment of the present invention. [Figure 9] This is a schematic operation flowchart of a system including a driving support device and a server device according to an embodiment of the present invention. [Figure 10] This is a configuration diagram of a plurality of model sets set in the server device according to an embodiment of the present invention. [Figure 11] This is a flowchart of a model set setting process by the server device according to an embodiment of the present invention. [Figure 12] This is a diagram showing two evaluation images according to an embodiment of the present invention. [Figure 13] This is a flowchart of a conversion data setting process by the server device according to an embodiment of the present invention. [Figure 14] This is a flowchart of an actual operation process by the driving support device according to an embodiment of the present invention. [Figure 15] This is a correspondence relationship diagram relating to a model according to an embodiment of the present invention. [Figure 16] This is a diagram showing two model sets and conversion data according to Example EX1 belonging to the embodiment of the present invention. [Figure 17] This is a flowchart of an integration process according to Example EX1 belonging to the embodiment of the present invention. [Figure 18] This is a relationship diagram between the object detection area of each object detection data and the object detection area of inference result data according to Example EX1 belonging to the embodiment of the present invention. [Figure 19] This is a diagram showing conversion data according to Example EX2 belonging to the embodiment of the present invention. [Figure 20] This is a diagram showing two model sets and conversion data according to Example EX3 belonging to the embodiment of the present invention. [Figure 21] This is a flowchart of an integration process according to Example EX3 belonging to the embodiment of the present invention. [Mode for Carrying Out the Invention]

[0009] Hereinafter, examples of embodiments of the present invention will be specifically described with reference to the drawings. In each of the referenced drawings, the same parts are denoted by the same reference numerals, and redundant descriptions relating to the same parts are omitted as a general rule. In addition, in this specification, for the sake of simplification of the description, symbols or reference numerals that refer to information, signals, physical quantities, functional parts, circuits, elements, or components may be indicated, and the names of the information, signals, physical quantities, functional parts, circuits, elements, or components corresponding to such symbols or reference numerals may be omitted or abbreviated.

[0010] Figure 1 shows the relationship between the vehicle V1 and other components assumed in an embodiment of the present invention. Vehicle V1 is any type of vehicle. Here, vehicle V1 is assumed to be an automobile or the like that travels on a road surface. A driver assistance device 10 is installed in vehicle V1. Person U1 is the user of the driver assistance device 10 and will be referred to as user U1 below. User U1 is an occupant of vehicle V1. In this embodiment, it is assumed that user U1 is the driver of vehicle V1. Hereafter, when simply referred to as "driver," it will refer to the driver of vehicle V1 (and therefore user U1). However, user U1 may be an occupant other than the driver.

[0011] Vehicle V1 is equipped with multiple electrical components, and these components form a driver assistance system that supports the user U1 in driving vehicle V1. Each electrical component forming the driver assistance system is capable of sending and receiving arbitrary signals and information to and from each other through an in-vehicle network formed within vehicle V1. The in-vehicle network includes, for example, CAN (Controller Area Network) and AVCLAN (Audio Visual Communication Local Area Network). The above multiple electrical components include a driver assistance device 10 and a camera CM. The driver assistance device 10 may be an information terminal device (hereinafter referred to as a user terminal device) brought into vehicle V1 by user U1. The user terminal device is a smartphone or tablet terminal, etc. The camera CM is installed in an appropriate location on vehicle V1. The camera CM has a shooting area set outside vehicle V1 and generates a camera image by capturing the scene within the shooting area. The camera image is the image of the shooting area captured by the camera CM. Data representing the content of any image is referred to as image data.

[0012] The camera CM periodically captures images at a predetermined frame rate and sequentially supplies the image data of the camera images to the driver assistance device 10 via the in-vehicle network. The camera CM may be a front camera, a rear camera, or a side camera. The shooting area of ​​the front camera is the area outside the vehicle V1 and includes the area in front of the vehicle V1. The shooting area of ​​the rear camera is the area outside the vehicle V1 and includes the area behind the vehicle V1. The shooting area of ​​the side camera is the area outside the vehicle V1 and includes the area to the side of the vehicle V1. Two or more cameras from the front camera, rear camera, and side camera may be installed on the vehicle V1 as the camera CM, but in Figure 1, it is assumed that the camera CM is a front camera.

[0013] Figure 2 shows an internal block diagram of the driver assistance system 10. The driver assistance system 10 comprises an in-vehicle controller 11, a memory 12, a communication unit 13, and an HMI 14.

[0014] The in-vehicle controller 11 is equipped with a processing unit as a hardware resource, including a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit). This processing unit may include an NPU (Neural Processing Unit). The in-vehicle controller 11 may implement any functions, operations, and processes to be realized by the in-vehicle controller 11 by executing a program recorded in memory 12 or any other recording medium (not shown). All or part of the operations mainly performed by the in-vehicle controller 11 shown below may be understood as operations performed by the processing unit within the in-vehicle controller 11.

[0015] Memory 12 is comprised of non-volatile memory such as ROM (Read-only memory) or flash memory, and volatile memory such as RAM (Random Access Memory). Memory 12 stores various data referenced by the in-vehicle controller 11, as well as various programs to be executed by the in-vehicle controller 11. The contents of memory 12 include model data 121 and conversion data 122. Model data 121 is data that defines the structure and operation of the image recognition model, which will be described later. Conversion data 122 will be described later.

[0016] The communication unit 13 is a communication circuit (communication module) that transmits and receives arbitrary signals with the other device. The other device for the communication unit 13 includes any device connected to the in-vehicle network and the external network (excluding the driver assistance device 10). The external network includes mobile communication networks, the internet, and intranets. The communication unit 13 may also be a communication circuit (communication module) located outside the driver assistance device 10. The communication circuit located outside the driver assistance device 10 may be a device installed in the vehicle V1 and a communication circuit shared by devices other than the driver assistance device 10 and the driver assistance device 10. The communication unit 13 may also be a communication circuit (communication module) located in the user terminal device. The in-vehicle controller 11 can transmit and receive arbitrary information with the other device using the communication unit 13, but the description of the communication unit 13 may be omitted below.

[0017] The HMI 14 is a Human-Machine Interface, an interface for mutual information transmission between the driver assistance device 10 (in other words, the in-vehicle controller 11) and the user U1. User U1 and the driver assistance device 10 (in other words, the in-vehicle controller 11) can mutually transmit arbitrary information through the HMI 14. The HMI 14 includes a microphone 141, a speaker 142, and a display device 143. The microphone 141 picks up ambient sounds at its installation location and converts them into electrical signals, which are audio signals. The speech of user U1 is included in the content picked up by the microphone 141. The speaker 142 outputs arbitrary sounds under the control of the in-vehicle controller 11. The display device 143 has a display screen such as a liquid crystal display panel and displays arbitrary images under the control of the in-vehicle controller 11. The display device 143 is equipped with a touch panel. The in-vehicle controller 11 can output arbitrary information to user U1 using the speaker 142 or the display device 143.

[0018] The in-vehicle controller 11 has multiple image recognition models and can perform inference processing using multiple image recognition models. In the inference processing, inference is performed to recognize objects in the camera image. Each image recognition model corresponds to an object detector, and therefore each image recognition model detects the type, position, and size of objects in the camera image. The in-vehicle controller 11 constructs each image recognition model based on the model data 121 in the memory 12.

[0019] Figure 3(a) shows a functional block diagram of an in-vehicle controller 11 focusing on image recognition. The in-vehicle controller 11 has functional blocks 111 to 113. Each function of functional blocks 111 to 113 may be realized by executing a program recorded in memory 12 or other arbitrary recording medium (not shown) on the in-vehicle controller 11. Functional block 111 is the image acquisition unit. The in-vehicle controller 11 is provided with multiple functional blocks 112, each functional block 112 being an image recognition model. Functional block 113 is the integration unit. The in-vehicle controller 11 is provided with m image recognition models 112, where m is any integer greater than or equal to 2. m is sometimes expressed as the number of models. As shown in Figure 3(b), the m image recognition models 112 consist of image recognition models 112[1] to 112[m], and when it is necessary to distinguish between multiple image recognition models 112, the symbols “112[1] to 112[m]” are referred to as appropriate. Any one of the m image recognition models 112 can be denoted as image recognition model 112[i], where i is an arbitrary integer, and in image recognition model 112[i], i represents any natural number less than or equal to m. The m image recognition models 112 and the integration unit 113 constitute an inference block 114, and the inference process described above is executed by the inference block 114. Hereafter, for the sake of simplification of the description, image recognition model 112 and image recognition model 112[i] will often be abbreviated as model 112 and model 112[i].

[0020] The multiple models 112 in the in-vehicle controller 11 are all different image recognition models. For example, model 112[1] is a trained AI model formed by machine learning a DNN (Deep Neural Network) using a first training dataset, and model 112[2] is a trained AI model formed by machine learning a DNN using a second training dataset. The first and second training datasets are different from each other. Furthermore, the machine learning conditions for forming model 112[1] may differ from the machine learning conditions for forming model 112[2]. The same applies to any combination of two models 112 that is different from the combination of models 112[1] and 112[2]. In this embodiment, AI refers to Artificial Intelligence.

[0021] The image acquisition unit 111 acquires a camera image from the camera CM and supplies an input image IN based on the camera image to each model 112. The image acquisition unit 111 may supply an image obtained by applying predetermined signal processing to the camera image as the input image IN to each model 112. However, the camera image itself may also be the input image IN.

[0022] The in-vehicle controller 11 can recognize (in other words, detect) objects present in the input image IN using each model 112. In any image recognition model, the object to be recognized is referred to as the object to be recognized. Each model 112 performs object detection processing based on the input image IN. Object detection processing may also be simply referred to as object detection. In each model 112, the object detection processing detects the object to be recognized in the input image IN. That is, in any image recognition model, the object detection processing sets the region in the input image IN where it is determined that the object to be recognized exists as the object detection region. In this embodiment, the setting of the object detection region and the detection of the object detection region are synonymous and may be interpreted interchangeably.

[0023] The information derived in the object detection process will be explained with reference to Figures 3(c) and 4. In Figure 4, image 600 represents an arbitrary two-dimensional image. When focusing on the object detection process using Model 112, the two-dimensional image 600 is the input image IN. Assuming a two-dimensional plane in which the two-dimensional image 600 is defined, we define two axes parallel to this two-dimensional plane, the X axis and the Y axis. The X axis and the Y axis are orthogonal to each other. The X axis is parallel to the horizontal direction of the two-dimensional image 600, and the Y axis is parallel to the vertical direction of the two-dimensional image 600. Hereafter, the two-dimensional plane parallel to the X axis and the Y axis will be referred to as the XY coordinate plane. The two-dimensional image 600 consists of multiple pixels arranged in a matrix on the XY coordinate plane (arranged in the X axis and Y axis directions, respectively).

[0024] In Figure 4, the dashed rectangular area labeled "DR" corresponds to the object detection area. The object detection area is generally referred to as the bounding box. For the sake of clarity, the object detection area will be referred to as the object detection area DR, and the data used to identify the position and shape of the object detection area will be explained. However, the information provided for the object detection area DR applies to any object detection area.

[0025] The object detection region DR is a part of the entire image area of ​​the two-dimensional image 600, and the position and shape of the object detection region DR are determined by the origin coordinates P. DR , width W DR and height H DR This is determined by [the specified method]. Therefore, the shape of the object detection area DR is a concept that represents not only the aspect ratio of the object detection area DR but also its size. The object detection area DR has a rectangular outline. The four sides of the rectangle are parallel to the X-axis or Y-axis, and the length of the side parallel to the X-axis is the width W. DR The length of the side parallel to the Y-axis is the height H. DR Here, the X-axis direction is considered the width direction, and the Y-axis direction is considered the height direction. That is, the length in the X-axis direction of the object detection region DR is the width W. DR Therefore, the length in the Y-axis direction in the object detection region DR is the height H. DRis. Among the vertices of the rectangle as the outer shape of the object detection region DR, the coordinates (coordinates on the XY coordinate plane) of the vertex located on the negative side of the X-axis and the negative side of the Y-axis are the origin coordinate P DR is. The object detection region DR is defined by the origin coordinate P DR and is a rectangular region extending to the positive side of the X-axis and the positive side of the Y-axis when viewed from the origin coordinate P.

[0026] Consider a case where a two-dimensional image 600 is supplied as an input image IN to an arbitrary model 112[i]. In this case, the model 112[i] performs object detection processing on the two-dimensional image 600, and generates and outputs object detection data as information representing the result of the object detection processing. When a recognition target object is detected from the two-dimensional image 600 in the object detection processing of the model 112[i], class data indicating the type of the detected recognition target object, position and shape data regarding the detected recognition target object, and a reliability representing the certainty of the result of the object detection processing are derived by the model 112[i]. That is, the object detection data generated and output when a recognition target object is detected from the two-dimensional image 600 by the object detection processing includes the class data, the position and shape data, and the reliability. The position and shape data derived by the object detection processing of the model 112[i] is data that specifies the position and shape of the object detection region DR for the recognition target object detected by the model 112[i] from the two-dimensional image 600, and is defined by the origin coordinate P DR , width W DR and height H DR . The reliability in the object detection data represents the certainty of the detection result that the object in the object detection region DR is the object indicated by the class data, and is generally referred to as a "confidence score". The higher the certainty, the greater the reliability. The reliability has a value of 0 or more and 1 or less. It can also be said that the reliability is the certainty of the detected class data and the object detection region DR.

[0027] The two-dimensional image 610 in Figure 5 is an example of a two-dimensional image 600. The two-dimensional image 610 contains an image of a vehicle 612. Consider the case where a vehicle is included in the objects to be recognized by model 112[i], and the two-dimensional image 610 is supplied to model 112[i] as the input image IN. In this case, model 112[i] performs object detection processing on the two-dimensional image 610, thereby setting the object detection region 613 as the object detection region DR. The object detection region 613 is the region in the two-dimensional image 610 where the image of the vehicle 612 exists. In this case, model 112[i] generates and outputs object detection data corresponding to the two-dimensional image 610, including class data indicating that the detected object is a "vehicle", position and shape data of the object detection region 613, and confidence level related to the detection of the object detection region 613.

[0028] Furthermore, when a two-dimensional image 600 is supplied to model 112[i] as the input image IN, and no image of the object to be recognized exists in the two-dimensional image 600, the object detection process by model 112[i] will not detect the object to be recognized from the two-dimensional image 600 (however, false detection is possible). When a two-dimensional image 600 is supplied to model 112[i] as the input image IN, and an image of the object to be recognized exists in the two-dimensional image 600, it is expected that the object to be recognized will be detected from the two-dimensional image 600 during the object detection process by model 112[i] (however, it may not be detected). When the object to be recognized is not detected from the input image IN during the object detection process by model 112[i], the object detection region is not set in model 112[i], and model 112[i] generates and outputs object detection data of "0". The object detection data of "0" is data that does not include class data, position and shape data, and confidence level, and indicates that the object to be recognized was not detected. In the following, unless otherwise specified, we will consider the case in which the object to be recognized is detected from the input image IN during object detection processing using each model 112.

[0029] There can be multiple types of objects to be recognized. When the types of objects to be recognized include the first to third types, the objects to be recognized for the first, second, and third types are, for example, "people," "vehicles," and "traffic lights," respectively. As for people, pedestrians located on roads etc. are mainly assumed. As for vehicles, vehicles that travel on the road surface, such as automobiles, are mainly assumed. As for traffic lights, they are traffic signals. The objects to be recognized in models 112[1] to 112[m] may be exactly the same. That is, when considering "m=2", for example, the objects to be recognized in model 112[1] and the objects to be recognized in model 112[2] may both be "people," "vehicles," and "traffic lights." Alternatively, the objects to be recognized in models 112[1] to 112[m] may be partially common. That is, when considering “m=2”, for example, the objects to be recognized in model 112[1] are “people,” “vehicles,” and “traffic lights,” while the objects to be recognized in model 112[2] may be only “people” and “vehicles,” or they may be “people,” “vehicles,” and “road signs.” However, unless otherwise specified below, the objects to be recognized in models 112[1] to 112[m] are considered to be exactly the same.

[0030] The object detection data derived in each model 112 is supplied to the integration unit 113. As shown in Figure 3(b), the object detection data derived in model 112[i] is sometimes specifically referred to as object detection data DET[i]. The integration unit 113 is supplied with object detection data DET[1] to DET[m]. The integration unit 113 generates inference result data by performing an integration process that combines the object detection data DET[1] to DET[m]. In other words, the inference block 114 performs inference (inference processing) to recognize the target object in the input image IN by integrating the object detection data DET[1] to DET[m] in the integration unit 113. The integration unit 113 outputs the inference result data as data indicating the result of the inference (in other words, data indicating the result of the inference processing).

[0031] Figure 3(d) shows the structure of the inference result data. The inference result data, like the object detection data, includes class data, position / shape data, and confidence level. The class data in the inference result data indicates the type of object to be recognized within the input image IN and is derived by the inference of inference block 114. The position / shape data in the inference result data identifies the position and shape of the object detection region DR for the object to be recognized and is derived by the inference of inference block 114. Similar to the position / shape data in the object result data, the position / shape data in the inference result data also includes the origin coordinate P. DR , width W DR and height H DR The position and shape of the object detection region DR are represented here. The confidence level in the inference result data is data that represents the likelihood of the detection result that the object in the object detection region DR is the object indicated in the class data, and also represents the likelihood of the inference by inference block 114. The confidence level in the inference result data, like the confidence level in the object detection data, has a value between 0 and 1.

[0032] In the following explanation, the object detection region of the inference result data refers to the object detection region whose position and shape are identified by the position and shape data in the inference result data (the region in which the inference block 114 determines that a target object exists). Similarly, the object detection region of the object detection data refers to the object detection region whose position and shape are identified by the position and shape data in the object detection data (the region in which the corresponding image recognition model determines that a target object exists).

[0033] The concept of the integrated processing is illustrated with reference to Figure 6. In the following, it is assumed that the class data derived in any image recognition model is represented by a string (i.e., a string indicating the name of the object to be recognized detected in the object detection process), and that this string is an English string. Figure 6 assumes the first reference case. In the first reference case, "m=2", the input image IN includes an image of vehicle 660, and the object to be recognized by each model 112 includes a vehicle. In the first reference case, model 112[1] generates object detection data including class data representing the string "automobile" and position and shape data that specifies the position and shape of the object detection region 661 through object detection processing on the input image IN. The object detection region 661 is data (bounding box) that specifies the position and shape of the object detection region DR for vehicle 660 detected by model 112[1] from the input image IN. In the first reference case, Model 112[2] generates object detection data including class data representing the string "automobile" and position and shape data specifying the position and shape of the object detection region 662 by performing object detection processing on the input image IN. The object detection region 662 is data (bounding box) that specifies the position and shape of the object detection region DR for the vehicle 660 detected by Model 112[2] from the input image IN. In the first reference case in Figure 6, the confidence level of each object detection data is assumed to be sufficiently high.

[0034] In the first reference case, the integration unit 113 generates inference result data by integrating the object detection data from models 112[1] and 112[2]. In the first reference case, since the class data in the object detection data of models 112[1] and 112[2] match, the integration unit 113 sets the integrated region of object detection regions 661 and 662 as object detection region 663. In the first reference case, object detection region 663 is the object detection region of the inference result data. In the first reference case, object detection region 661 includes only a part of the image of vehicle 660, and object detection region 662 includes only the other part of the image of vehicle 660. Therefore, by integrating object detection regions 661 and 662, it is possible to set an object detection region 663 that accurately represents the position and size of vehicle 660 in the input image IN.

[0035] Next, refer to Figure 7. Figure 7 assumes a second reference case. Similar to the first reference case, in the second reference case, “m=2”, the input image IN includes an image of vehicle 660, and the object to be recognized by each model 112 includes a vehicle. In the second reference case, model 112[1] generates object detection data including class data representing the string “automobile” and position and shape data specifying the position and shape of the object detection region 661 through object detection processing on the input image IN. In the second reference case, model 112[2] generates object detection data including class data representing the string “car” and position and shape data specifying the position and shape of the object detection region 662 through object detection processing on the input image IN. In the second reference case, object detection regions 661 and 662 are object detection regions for the common vehicle 660, but the strings of the class data derived by models 112[1] and 112[2] are different. In the second reference case, a reference device (not shown) different from the driver assistance device 10 cannot perform the integration process due to differences in class data. In contrast, the integration unit 113 related to the driver assistance device 10 can perform the same integration process as in the first reference case by converting the class data using the conversion data 122 (see Figure 2) (Figure 7 does not show how this integration process is performed).

[0036] The converted data 122 is generated and set by the server device 20 shown in Figure 8. Figure 8 is an internal block diagram of the server device 20. The server device 20 is an information processing device composed of one or more computer devices. The server device 20 may be formed using cloud computing. The server device 20 includes a server controller 21, memory 22, and communication unit 23. An HMI having a configuration equivalent to the HMI 14 described above may be provided in the server device 20, in which case the operator of the server device 20 can input various operations and instructions to the server device 20 through the HMI of the server device 20.

[0037] The server controller 21 is equipped with a processing unit including a CPU and a GPU as hardware resources. This processing unit may include an NPU (Neural Processing Unit). The server controller 21 may implement any functions, operations, and processes to be realized by the server controller 21 by executing programs recorded in memory 22 or other arbitrary recording media (not shown). All or part of the operations mainly performed by the server controller 21 described below may be understood as operations performed by the processing unit within the server controller 21. Memory 22 is composed of non-volatile memory such as ROM or flash memory and volatile memory such as RAM. Memory 22 stores various data referenced by the server controller 21, as well as various programs to be executed by the server controller 21. The communication unit 23 is a communication circuit (communication module) that sends and receives arbitrary signals with the other device. The other device for the communication unit 23 includes any device connected to the above-mentioned external network. The server controller 21 can send and receive arbitrary information with the other device using the communication unit 23, but the description of the communication unit 23 may be omitted below.

[0038] Figure 9 is a schematic operation flowchart of a system comprising a driver assistance device 10 and a server device 20. The system comprising the driver assistance device 10 and the server device 20 can be called an image recognition system, and in this image recognition system, steps S1, S2, and S3 are executed sequentially. In step S1, the server device 20 executes the model set setting process. In the model set setting process, the server controller 21 generates and sets m model sets (details will be described later). After step S1, the system proceeds to step S2. In step S2, the server device 20 executes the conversion data setting process. In the conversion data setting process, the server controller 21 generates and sets the above-mentioned conversion data 122 (details will be described later). After step S2, the system proceeds to step S3. In step S3, the driver assistance device 10 executes the actual operation process. All operations of the driver assistance device 10 are operations in the actual operation process. The actual operation process is executed with the conversion data 122 set in the conversion data setting process stored in the memory 12 of the driver assistance device 10 (i.e., the driver assistance device 10 operates).

[0039] Figure 10 shows the structure of the m model sets generated and configured in the model set configuration process. Each model set is formed within the server controller 21 during the model set configuration process. Each model set has a configuration in which header data is added to a trained AI model. Therefore, each model set has a trained AI model and header data. The m model sets are denoted as model sets MS[1] to MS[m]. The header data of model set MS[i] is sometimes specifically referred to as header data HD[i]. The trained AI model of model set MS[i] is sometimes specifically referred to as trained AI model MDL[i], and is sometimes abbreviated as model MDL[i].

[0040] The trained AI model in each model set is an image recognition model that performs object detection processing, and therefore can also be called an object detector. The same m models as models MDL[1] to MDL[m] are incorporated into the driver assistance device 10 as models 112[1] to 112[m] during the actual operation process. For any integer i satisfying "1 ≤ i ≤ m", model MDL[i] is a trained AI model formed by machine learning a DNN using the i-th training dataset. The machine learning method for forming the image recognition model (object detector) is well known, so the explanation of that method is omitted. The first to m training datasets are all different from each other. Also, the machine learning conditions for forming model MDL[1] may differ from the machine learning conditions for forming model MDL[2]. The same applies to any combination of two models different from the combination of models MDL[1] and MDL[2]. Each training dataset has a large number of training images. Each training image is a two-dimensional image containing images of various types of objects to be recognized.

[0041] [Model Set Setup Process] Referring to Figure 11, and with reference to Figure 10 as appropriate, the flow of the model set setting process will be explained. Figure 11 is a flowchart of the model set setting process in step S1 of Figure 9. In the model set setting process, the process in step S111 is performed first. In step S111, the server controller 21 acquires m trained AI models as models MDL[1] to MDL[m]. At the time of this application, many trained AI models capable of performing object detection processing have been put into practical use. The server controller 21 can acquire and use an existing trained AI model as model MDL[i]. However, the server controller 21 itself may generate and acquire model MDL[i] by performing machine learning based on the i-th training dataset. After step S111, the process proceeds to step S112.

[0042] In step S112, the server controller 21 sets the variable j, which it manages, to 1. After step S112, the process proceeds to step S113. There are multiple objects to be recognized by the m image recognition models, Model MDL[1] to MDL[m]. The objects to be recognized by Model MDL[i] include n different types of objects. The n types of objects to be recognized consist of objects of type 1 to n. In other words, objects of type 1 to n are objects of type 1 to n. n represents any integer greater than or equal to 2. n is sometimes expressed as the number of types of objects to be recognized. The number of types of objects n can be different among Model MDL[1] to MDL[m]. That is, the number of types of objects n in any one of the Model MDL[1] to MDL[m] models can be different from the number of types of objects n in any other Model MDL[1] to MDL[m]. However, unless otherwise specified, the number of types of objects to be recognized, n, is assumed to be common between models MDL[1] to MDL[m]. In step S113, the server controller 21 sets the evaluation image E[j]. The evaluation image E[j] is a two-dimensional image containing images of the j-th type of object. For example, if the first type of object is a "person" and the second type of object is a "vehicle", examples of evaluation images E[1] and E[2] in this case are shown in Figure 12.

[0043] The server controller 21 can extract and set an evaluation image E[j] from a large number of training images included in the first to m training datasets that contains an image of the j-th type of object. For example, the server controller 21 may randomly extract an evaluation image E[j] from a group of training images that contain an image of the j-th type of object, from among the training images included in the first to m training datasets. The server controller 21 may also set the evaluation image E[j] according to instructions from the operator of the server device 20. The evaluation image E[j] may be extracted from an evaluation image set different from the training dataset. The evaluation image set has a group of two-dimensional images that contain an image of the j-th type of object. In any case, an evaluation image E[j] containing an image of the j-th type of object is set in step S113. Here, the evaluation image E[j] is assumed to contain only an image of the j-th type of object among the first to n-th type of objects.

[0044] Each two-dimensional image in each training image and evaluation image set is, for example, an image taken by a camera installed on an arbitrary cooperating vehicle (not shown). The appearance of the j-th type object in the captured image will differ depending on the shooting scene (time of day, etc.) of the cooperating vehicle's camera or the positional relationship between the camera and the j-th type object. It is desirable that the captured image that shows the j-th type object in a representative manner be adopted as the evaluation image E[j]. After step S113, proceed to step S114.

[0045] In step S114, the server controller 21 supplies the evaluation image E[j] to each of the models MDL[1] to MDL[m], causing each of the models MDL[1] to MDL[m] to perform object detection processing on the evaluation image E[j]. As a result, each of the models MDL[1] to MDL[m] detects an object of type j in the evaluation image E[j]. In step S114, each of the models MDL[1] to MDL[m] generates object detection data that shows the result of the object detection processing. The m object detection data generated by each of the models MDL[1] to MDL[m] each has class data, position and shape data, and confidence level. In step S114, the evaluation image E[j] corresponds to the input image IN for the models MDL[1] to MDL[m].

[0046] The significance of class data, position / shape data, and confidence score in object detection data is as described above. Therefore, in the object detection data generated by model MDL[i] in step S114, the class data is a string indicating the type of object to be recognized in the evaluation image E[j], and thus a string indicating the name of the j-th type object (e.g., "automobile"). In the object detection data generated by model MDL[i] in step S114, the position / shape data is data that identifies the position and shape of the object detection region for the object to be recognized (the j-th type object) detected by model MDL[i] from the evaluation image E[j]. In the object detection data generated by model MDL[i] in step S114, the confidence score represents the likelihood of the detection result that the object in the object detection region is the object indicated by the class data, and is generally called the "confidence score". The higher the likelihood, the greater the confidence score. As described above, the confidence score has a value of 0 or greater and 1 or less. After step S114, proceed to step S115.

[0047] In step S115, the server controller 21 determines that all m confidence levels derived from the models MDL[1] to MDL[m] for the evaluation image E[j] are within a predetermined threshold TH. CS Determine if it is above the threshold TH. CSThe confidence level is a positive value less than 1 but close to 1, for example, 0.90 or 0.95. CS If the above is true (Y in step S115), proceed to step S116, and if at least one of the above m confidence values ​​is a threshold TH CS If the confidence level is less than N (N in step S115), the process proceeds to step S117. In step S117, the server controller 21 discards the object detection data derived in the preceding step S114. After step S117, the process returns to step S113. If the process returns to step S113 via step S117, the evaluation image E[j] is reset in step S113. In resetting the evaluation image E[j], an image different from the image previously set as evaluation image E[j] and containing an image of the j-th type of object is set as evaluation image E[j]. After the evaluation image E[j] is reset, the processes in steps S114 and S115 are performed again.

[0048] In step S116, the server controller 21 determines whether "j=n" is true or false. If "j=n" is true (Y in step S116), the process proceeds to step S119; if "j=n" is false (N in step S116), the process proceeds to step S118. In step S118, the server controller 21 adds 1 to the variable j. After step S118, the process returns to step S113.

[0049] At step S119, object detection data from models MDL[1] to MDL[m] has been derived for each evaluation image, and the confidence level in each of the derived object detection data is the threshold TH. CS That concludes the explanation. In other words, by the time we proceed to step S119, object detection data with sufficiently high confidence has been derived for each evaluation image by models MDL[1] to MDL[m]. The class data in the object detection data derived by model MDL[i] for evaluation image E[j] is specifically called class data C[i,j]. The confidence level in the object detection data including class data C[i,j] is the threshold TH CSThat concludes the explanation. The class data C[i,j] can also be referred to as the evaluation class data.

[0050] In step S119, the server controller 21 sets the variable i, which it manages, to 1. After step S119, the process proceeds to step S120. In step S120, the server controller 21 sets a model set MS[i] which has header data HD[i] and model MDL[i]. In step S120, the server controller 21 stores evaluation images E[1]~E[n] and class data C[i,1]~C[i,n] in the header data HD[i]. At this time, the server controller 21 associates evaluation image E[j] and class data C[i,j] with each other for any integer j that satisfies "1≦j≦n". Therefore, if "n=2", the header data HD[i] will store a pair of evaluation image E[1] and class data C[i,1], and a pair of evaluation image E[2] and class data C[i,2]. If "n=3", then in addition to those set data, the set data of the evaluation image E[3] and class data C[i,3] is stored in the header data HD[i]. The same applies when "n≧4". The server controller 21 may either keep the configured model set MS[i] itself or have it kept in memory 22. After step S120, proceed to step S121.

[0051] In step S121, the server controller 21 determines whether "i=m" is true or false. If "i=m" is true (Y in step S121), the model set setting process shown in Figure 11 is completed. If "i=m" is false (N in step S121), the process proceeds to step S122. In step S122, the server controller 21 adds 1 to the variable i. After step S122, the process returns to step S120.

[0052] At step S121, when "i=m" is satisfied, models MS[1] to MS[m] having the structure shown in Figure 10 have already been set up in the server controller 21.

[0053] The process consisting of steps S113 to S115 can be called a unit process. The server controller 21 performs unit processing on each of the evaluation images E[1] to E[n]. In the unit processing for evaluation image E[j], the server controller 21 inputs evaluation image E[j] to model MDL[1] and causes model MDL[1] to perform object detection processing, thereby deriving class data C[1,j] from model MDL[1]. Similarly, in the unit processing for evaluation image E[j], the server controller 21 inputs evaluation image E[j] to model MDL[2] and causes model MDL[2] to perform object detection processing, thereby deriving class data C[2,j] from model MDL[2]. The same applies to models MDL[3] to MDL[m]. However, the confidence deriven in each unit process is always the threshold TH. CS The above is the premise.

[0054] The server controller 21 sets up model sets MS[1] to MS[m] by performing unit processing on each of the evaluation images E[1] to E[n] (steps S113 to S122). Model set MS[i] has header data HD[i] and model MDL[i]. Header data HD[1] has a pair of data for each evaluation image consisting of an evaluation image and class data (evaluation class data). The pair of evaluation image E[1] and class data C[1,1] is an example of a pair of data in header data HD[1], and the pair of evaluation image E[2] and class data C[1,2] is another example of a pair of data in header data HD[1]. Similarly, header data HD[2] has a pair of data for each evaluation image consisting of an evaluation image and class data (evaluation class data). The pair data of evaluation image E[1] and class data C[2,1] is one example of a pair data in header data HD[2], and the pair data of evaluation image E[2] and class data C[2,2] is another example of a pair data in header data HD[2]. The same applies to header data HD[3] to HD[m].

[0055] [Conversion Data Setting Process] Figure 13 is a flowchart of the conversion data setting process in step S2 of Figure 9. In the conversion data setting process, the process in step S211 is performed first. In step S211, the server controller 21 sets the variable j that it manages to 1. After step S211, the process proceeds to step S212.

[0056] In step S212, the server controller 21 identifies the class correspondence for the j-th type of object based on m class data C[1,j] to C[m,j] for the j-th type of object. The class correspondence for the j-th type of object is called the j-th class correspondence. The j-th class correspondence shows the correspondence between the m class data C[1,j] to C[m,j] mentioned above. For example, if "m=2", the first class correspondence showing the correspondence between class data C[1,1] and C[2,1] is identified based on two class data C[1,1] and C[2,1] for the first type of object. If "m=2", the second class correspondence showing the correspondence between class data C[1,2] and C[2,2] is identified based on two class data C[1,2] and C[2,2] for the second type of object. The same applies to the third type of object to the nth type of object. After step S212, the process proceeds to step S213.

[0057] In step S213, the server controller 21 determines whether "j=n" is true or false. If "j=n" is true (Y in step S213), the process proceeds to step S215. If "j=n" is false (N in step S213), the process proceeds to step S214. In step S214, the server controller 21 adds 1 to the variable j. After step S214, the process returns to step S212.

[0058] By the time the process proceeds to step S215, the correspondence between the first to the nth class has been identified through the processing of step S212, which has been executed n times. In step S215, the server controller 21 generates and sets conversion data based on the correspondence between the first to the nth class. This conversion data is stored in the server controller 21 or in memory 22. The conversion data setting process is completed with the setting of the conversion data in step S215.

[0059] As described above, the same m models as models MDL[1] to MDL[m] are incorporated into the driver assistance device 10 as models 112[1] to 112[m] during the actual operation process. The model data 121 in memory 12 (see Figure 2) is data that defines the structure and operation of models MDL[1] to MDL[m] (and therefore models 112[1] to 112[m]). The in-vehicle controller 11 constructs models 112[1] to 112[m] based on the model data 121. The conversion data generated and set in step S215 is held in the memory 12 of the driver assistance device 10 as conversion data 122 during the actual operation process. Even when multiple models 112 derive different class data from each other for a common recognition target object, as illustrated in the second reference case in Figure 7, conversion data that enables the integration of position and shape data is generated and set in step S215.

[0060] [Actual Operation Process] Figure 14 is a flowchart of the actual operation process in step S3 of Figure 9. For example, the actual operation process begins when the engine of vehicle V1 starts based on an operation on the ignition switch provided on vehicle V1. Once the actual operation process begins, the process in step S311 is executed first. During the execution period of the actual operation process, camera CM periodically takes pictures, sequentially generating camera images, and in step S311, the image acquisition unit 111 acquires the input image IN based on the latest camera image (see also Figures 3(a) and (b)). After step S311, the process proceeds to step S312.

[0061] In step S312, models 112[1] to 112[m] generate object detection data DET[1] to DET[m] by performing object detection processing on the latest input image IN. After step S312, the process proceeds to step S313.

[0062] In step S313, the integration unit 113 generates inference result data from object detection data DET[1] to DET[m] through integration processing. Class conversion processing based on conversion data 212 is performed during the integration processing (a specific example will be described later). After step S313, the process proceeds to step S314.

[0063] In step S314, the in-vehicle controller 11 performs inference result processing based on the inference result data. For example, if the inference result data indicates that a person is present in the direction of travel of vehicle V1, the in-vehicle controller 11 involved in the inference result processing uses the HMI 14 to issue a warning notification to user U1. The warning notification informs user U1 that a person is present in the direction of travel of vehicle V1 and is implemented by outputting sound from speaker 142, displaying an image on display device 143, or both. Also, for example, if the inference result data indicates that a vehicle is present in the direction of travel of vehicle V1, the in-vehicle controller 11 involved in the inference result processing uses the HMI 14 to issue a notification regarding the distance between vehicles. The notification regarding the distance between vehicles informs user U1 that an appropriate distance should be maintained between vehicle V1 and other vehicles in the direction of travel of vehicle V1 and is implemented by outputting sound from speaker 142, displaying an image on display device 143, or both. Furthermore, the in-vehicle controller 11 may perform appropriate vehicle control, such as reducing the speed of vehicle V1, along with issuing a warning notification or a notification regarding the distance between vehicles. After step S314, the process returns to step S311.

[0064] In the following, for the sake of explanation, we will introduce the terms Model A, B, and C, as shown in Figure 15. When the number of models m is 3 or more, Models A, B, and C are provided in the server device 20 and the driver support device 10, respectively. Models A, B, and C in the server device 20 are Models MDL[1], MDL[2], and MDL[3], respectively (see Figure 10). Models A, B, and C in the driver support device 10 are Models 112[1], 112[2], and 112[3], respectively (see Figure 2). When the number of models m is 2, Models A and B are provided in the server device 20 and the driver support device 10, respectively, and Model C does not exist. When the number of models m is 2, the description of Model C is ignored.

[0065] Furthermore, the class data in the object detection data derived by Model A is referred to as the output class data of Model A. The class data in the object detection data derived by Model B is referred to as the output class data of Model B. The class data in the object detection data derived by Model C is referred to as the output class data of Model C. Similarly, the class data in the object detection data derived by Model 112[i] is referred to as the output class data of Model 112[i]. Similarly, the class data in the object detection data derived by Model MDL[i] is referred to as the output class data of Model MDL[i]. For example, in the first reference case in Figure 6, the output class data of Models 112[1] and 112[2] both represent the string "automobile". Also, for example, in the second reference case in Figure 7, the output class data of Models 112[1] and 112[2] represent the strings "automobile" and "car", respectively.

[0066] In the following practical examples, specific operational examples, application techniques, modification techniques, etc., related to the apparatus and system of the present invention will be described. In this embodiment, unless otherwise specified and without contradiction, the matters described above apply to each of the following embodiments. In the event of any contradiction between the matters described above and those described in each embodiment, the description in the respective embodiment may take precedence. Furthermore, unless there is a contradiction, the matters described in any of the embodiments shown below may be applied to any other embodiment (i.e., it is possible to combine any two or more embodiments from the multiple embodiments).

[0067] Multiple objects to be recognized may be detected by each model 112 from a single input image IN. However, in the embodiments described below, for the sake of concretization and simplification of the explanation, unless otherwise specified, it will be assumed that only one object to be recognized is detected by each model 112 from a single input image IN. Furthermore, in the embodiments described below, the IoU (Intersection over Union) for multiple object detection regions derived by multiple models 112 is sufficiently high (for example, a predetermined threshold TH). IOU (The above is true.) IoU is an index that represents the degree of overlap of multiple object detection regions, and the meaning and derivation method of IoU are publicly known. For example, the IoU between the first object detection region, which is the object detection region of object detection data DET[1], and the second object detection region, which is the object detection region of object detection data DET[2], is given by "IoU = S I [1:2] / S U [1:2]” is represented as S I [1:2] represents the area of ​​the intersection of the first and second object detection regions. That is, S I [1:2] represents the area where the first and second object detection regions overlap. U [1:2] represents the area of ​​the union of the first and second object detection regions. That is, S U [1:2] represents the sum of the area of ​​the region where the first and second object detection regions overlap, the area of ​​the region contained only within the first object detection region, and the area of ​​the region contained only within the second object detection region. The area of ​​any region on the image is represented by the total number of pixels belonging to that region.

[0068] <<Example EX1>> Example EX1 will now be described. Example EX1 assumes that the number of models m is 2 and the number of types of objects to be recognized n is also 2. Figure 16(a) shows an overview of Case CS1 assumed in Example EX1. In Case CS1, the first type of object (object to be recognized) is a person, and the second type of object (object to be recognized) is an automobile belonging to the vehicle category. Here, an automobile is assumed to be a passenger car, expressed as "car" in English. In Example EX1, evaluation image E[1] is a two-dimensional image containing the image of a person, which is the first type of object, and evaluation image E[2] is a two-dimensional image containing the image of a vehicle (automobile), which is the second type of object. Model MDL[1] for Case CS1 assigns the class data "person" to people and the class data "automobile" to vehicles (automobiles) during the object detection process. Model MDL[2] for Case CS1 assigns the class data "person" to people and the class data "car" to vehicles (automobiles) during the object detection process. Therefore, in the model set setting process for case CS1, the strings in class data C[1,1], C[1,2], C[2,1], and C[2,2] are set to "person", "automobile", "person", and "car", respectively. The English term "automobile" is a concept that includes automobiles.

[0069] The server controller 21 identifies a first class correspondence that shows the correspondence between class data C[1,1] and C[2,1], and a second class correspondence that shows the correspondence between class data C[1,2] and C[2,2] (step S212 in Figure 13). The first class correspondence indicates that for the first type of object, both models A and B assign the class data "person". The second class correspondence indicates that for the second type of object, model A assigns the class data "automobile" while model B assigns the class data "car". In this case, based on the second class correspondence, the server controller 21 sets the conversion data so that the output class data of model A and the output class data of model B match each other after conversion by the conversion data for the second type of object (step S215 in Figure 13).

[0070] Specifically, in case CS1, the conversion data 122_1 shown in Figure 16(b) is set in the server controller 21, and therefore the conversion data 122_1 is stored as conversion data 122 in the memory 12 of the driver assistance device 10. The integration unit 113 may convert the output class data of model 112[1] or 112[2] to other class data, and the conversion rules are defined in the conversion data 122. In arbitrary conversion data, the output class data of each model is individually classified as target class data or non-target class data (information indicating the content of the classification is included in the conversion data). In conversion data 122_1, the output class data of "person" by model A, the output class data of "automobile" by model A, and the output class data of "person" by model B are classified as non-target class data. In conversion data 122_1, the output class data of "car" by model B is classified as target class data. Arbitrary conversion data is data that specifies that only the output class data classified as target class data should be converted to other class data. The class data after this conversion is performed is called the integration class data. Therefore, conversion data 122_1 specifies that the output class data of "car" by Model B is converted to other class data (integration class data), and this other class data represents "automobile".

[0071] In case CS1, the converted data 122_1 is used as converted data 122 in the actual operation process. Based on the converted data 122_1 (according to the specifications in converted data 122_1), the integration unit 113 performs a class conversion process on the output class data of model 112[2] only if the output class data of model 112[2] is "car". In this class conversion process, the integration unit 113 converts the output class data of model 112[2] from "car" to "automobile" (i.e., converts it to other class data that indicates "automobile"). After this conversion, the output class data of models 112[1] and 112[2] match each other. Based on this match, the integration unit 113 sets the object detection region of the inference result data by integrating the object detection region of object detection data DET[1] and the object detection region of object detection data DET[2].

[0072] The integration unit 113 does not perform class conversion processing on output class data that is classified as non-target class data in the conversion data 122_1. In other words, output class data classified as non-target class data is maintained as is without being modified by the integration unit 113.

[0073] Figure 17 shows a flowchart of the integration process performed by the integration unit 113. The integration process in Figure 17 consists of steps S331 to S339. The integration process in Figure 17 can be performed in step S313 of Figure 14. That is, the inference result data set in the integration process in Figure 17 may be the inference result data generated in step S313. The integration process in Figure 17 starts from step S331.

[0074] In step S331, the integration unit 113 determines whether the output class data of any of the models 112 contains the target class data. If the output class data of any of the models 112 contains the target class data (Y in step S331), the process proceeds to step S332. If the output class data of any of the models 112 does not contain the target class data (N in step S331), the process proceeds to step S337.

[0075] In step S332, the integration unit 113 converts the target class data into integration class data, which is other class data, based on the conversion data 122 (in accordance with the provisions in the conversion data 122). After step S332, the process proceeds to step S333.

[0076] In step S333, the integration unit 113 determines whether the output class data of model 112[1], which is the non-target class data, matches the integration class data. In case CS1 assumed in embodiment EX1, the integration class data related to step S333 is the integration class data based on the output class data of model 112[2]. If the output class data of model 112[1], which is the non-target class data, matches the integration class data (Y in step S333), the process proceeds to step S334. If the non-target class data and the integration class data do not match (N in step S333), the process proceeds to step S339.

[0077] In step S334, the integration unit 113 sets the output class data of model 112[1], which is non-target class data, or the integration class data, as the class data in the inference result data. When proceeding to step S334, the output class data of model 112[1] and the integration class data are the same, so regardless of which is set as the class data in the inference result data, there is no difference in the class data in the inference result data. After step S334, the process proceeds to step S335.

[0078] In step S335, the integration unit 113 sets the object detection region of the inference result data by integrating the object detection regions of each object detection data. In Example EX1, it is assumed that the number of models m is 2, so in step S335, the integration unit 113 sets the object detection region of the inference result data by integrating the object detection region of object detection data DET[1] and the object detection region of object detection data DET[2]. In step S335, for example, the following first or second region integration method can be employed.

[0079] The integration unit 113 of the first domain integration method derives a composite region of the object detection region of object detection data DET[1] and the object detection region of object detection data DET[2], and sets the smallest rectangular region encompassing this composite region as the object detection region of the inference result data. Figure 18 shows the relationship between the object detection region 711 of object detection data DET[1], the object detection region 712 of object detection data DET[2], and the object detection region 713 of the inference result data when the first domain integration method is adopted. All object detection regions 711 to 713 are set within the input image IN, and the smallest rectangular region encompassing the composite region (union) of object detection regions 711 and 712 is the object detection region 713. The integration unit 113 of the second domain integration method sets the region where the object detection region of object detection data DET[1] and the object detection region of object detection data DET[2] overlap (intersection) as the object detection region of the inference result data. After step S335, proceed to step S336.

[0080] In step S336, the integration unit 113 sets the confidence level of the inference result data based on the confidence level in each object detection data. In Example EX1, it is assumed that the number of models m is 2, so in step S335, the integration unit 113 sets the confidence level in the inference result data based on the confidence level in the object detection data DET[1] and the confidence level in the object detection data DET[2]. In step S336, the integration unit 113 can, for example, set the confidence level in the inference result data using statistical values ​​(e.g., maximum value, minimum value, or average value) of the confidence level in the object detection data DET[1] and the confidence level in the object detection data DET[2]. When the processing in step S336 is performed, the integration process in Figure 17 is completed upon the execution of the processing in step S336.

[0081] In step S337, the integration unit 113 determines whether the output class data of each model 112 matches. If the output class data of each model 112 matches (Y in step S337), the process proceeds to step S338. If the output class data of each model 112 does not match (N in step S337), the process proceeds to step S339. In embodiment EX1, it is assumed that the number of models m is 2, so in step S337, the integration unit 113 determines whether the output class data of models 112[1] and 112[2] match. If the output class data of model 112[1] and the output class data of model 112[2] match (Y in step S337), the process proceeds to step S338. If the output class data of model 112[1] and the output class data of model 112[2] do not match (N in step S337), the process proceeds to step S339.

[0082] In step S338, the integration unit 113 sets the output class data of one of the models 112 as the class data in the inference result data. When proceeding to step S338, the output class data of each model 112 are identical, so regardless of which output class data is set as the class data in the inference result data, there is no difference in the class data in the inference result data. After step S338, the process proceeds to step S335.

[0083] In step S339, the integration unit 113 performs mismatch handling. In the mismatch handling, the integration unit 113 sets one of the object detection data as the inference result data. That is, for example, in the mismatch handling, the integration unit 113 sets either object detection data DET[1] or DET[2] as the inference result data. Which of the object detection data DET[1] or DET[2] is set as the inference result data may be predetermined by the integration unit 113. When the inference result data is set in step S339, the integration process shown in Figure 17 is completed. If the process proceeds to step S339, it is considered that there is an error in the object detection processing by one of the models 112. For this reason, the integration unit 113 involved in the mismatch handling may complete the integration process shown in Figure 17 without generating inference result data (i.e., it may stop generating inference result data). In this case, in the actual operational process shown in Figure 14, the transition to step S311 should occur without generating inference result data in step S313 and without performing the processing in step S314.

[0084] We will now explain the integration process in case CS1_a, assuming that the object to be recognized in the input image IN is a person. In case CS1_a, we assume that the object to be recognized was correctly detected by models 112[1] and 112[2]. In case CS1_a, the output class data of both models 112[1] and 112[2] is "person", and the output class data of "person" is not the target class data (see Figure 16(b)). Therefore, in case CS1_a, in the integration process, the output class data of model 112[1] or 112[2] is set as the class data of the inference result data, going from step S331 through step S337 to step S338. After that, the processes in steps S335 and S336 are executed.

[0085] We will now describe the integration process in case CS1_b, assuming that the object to be recognized in the input image IN is a vehicle. Figure 18 shows the outputs of models 112[1] and 112[2] and the integration unit 113 in case CS1_b. In case CS1_b, we assume that the object to be recognized was correctly detected by models 112[1] and 112[2]. In case CS1_b, the output class data of models 112[1] and 112[2] are "automobile" and "car", respectively. The output class data of "car" from model 112[2] is the target class data (see Figure 16(b)). Therefore, in case CS1_b, in the integration process, proceeding from step S331 to step S332, the output class data of model 112[2] is converted from "car" to "automobile" (i.e., converted to integration class data indicating "automobile"). Then, since the output class data ("automobile") of model 112[1], which corresponds to the non-target class data, matches the integration class data ("automobile"), the processing in steps S334 to S336 is executed via step S333.

[0086] We will now explain the integration process in case CS1_c, assuming that the object to be recognized in the input image IN is a person. However, in case CS1_c, we will assume that model 112[1] correctly detected the object in the input image IN as a person, but model 112[2] incorrectly detected the object as a vehicle. In this case, the output class data of models 112[1] and 112[2] are "person" and "car", respectively. The output class data of "car" from model 112[2] is the target class data (see Figure 16(b)). Therefore, in case CS1_c, in the integration process, proceeding from step S331 to step S332, the output class data of model 112[2] is converted from "car" to "automobile" (i.e., converted to integration class data indicating "automobile"). However, since the output class data ("person") of model 112[1], which corresponds to the non-target class data, does not match the integration class data ("automobile"), the process proceeds to step S339 via step S333, and the mismatch handling process in step S339 is executed.

[0087] We will now describe the integration process in case CS1_d, assuming that the object to be recognized in the input image IN is a person. However, in case CS1_d, model 112[2] correctly detected that the object in the input image IN was a person, but model 112[1] incorrectly detected that the object in the input image IN was a vehicle. In case CS1_d, the output class data of models 112[1] and 112[2] are "automobile" and "person," respectively, which are not the target class data (see Figure 16(b)) and do not match each other. Therefore, in case CS1_d, the integration process proceeds from step S331 through step S337 to step S339, and the mismatch handling process in step S339 is executed. Various other cases may occur, but the integration process is executed according to the flow shown in Figure 17.

[0088] Thus, the processing content of the integration process branches depending on whether the class data in the object detection data DET[2] corresponds to the target class data. Class data indicates the type of object, but for the class data in the object detection data DET[2] to correspond to the target class data is equivalent to the class data in the object detection data DET[2] indicating a specific type. More specifically, for the class data in the object detection data DET[2] to indicate a specific type is that the class data in the object detection data DET[2] indicates the name of an object of a specific type. In Example EX1, the name of an object of a specific type is "car". If the integration unit 113 determines that the class data in the object detection data DET[2] indicates a specific type, it converts the class data in the object detection data DET[2] to integration class data, which is other class data, according to a pre-set conversion rule (according to the conversion data 122) (S332). Then, when the class data in the object detection data DET[1] corresponding to the non-target class data matches the integration class data, the integration unit 113 executes the processing in steps S334 to S336.

[0089] The processing in step S335 enables the derivation of an object detection region that accurately represents the position and size of the object to be recognized in the input image IN during inference related to object detection. When the driver assistance device 10 is configured by combining multiple existing image recognition models, different class data are often derived from each other for a common object to be recognized among the multiple image recognition models. The driver assistance device 10 enables the derivation of a highly accurate object detection region even when different class data are derived from each other for a common object to be recognized among the multiple image recognition models through integrated processing using the converted data 122.

[0090] As described above, if the class data in the object detection data DET[2] indicates a specific type, the class data in the object detection data DET[2] is converted to integration class data of a different class according to a pre-set conversion rule (according to the conversion data 122) (S332). However, if the class data in the object detection data DET[1] corresponding to the non-target class data does not match the integration class data, the integration unit 113 performs a mismatch handling process.

[0091] If there is a mismatch between the class data in the object detection data DET[1] and the integration class data (the class data in the object detection data DET[2] after class conversion processing), it is considered that there is an error in the object detection processing of either model 112[1] or 112[2]. In such cases, it is not appropriate to integrate the object detection regions, and the mismatch handling process prevents the inappropriate integration of object detection regions.

[0092] Furthermore, there are cases where the class data in the object detection data DET[2] indicates a different type from the specified type (cases where it indicates an object name of a different type from the specified type), and examples of such cases are cases CS1_a and CS1_d. In cases where the class data in the object detection data DET[2] indicates a different type from the specified type, the process moves from step S331 to step S337, where the class data in the object detection data DET[1] and the class data in the object detection data DET[2] are compared. When the class data in the object detection data DET[1] and the class data in the object detection data DET[2] match (Y in step S337), the processes in steps S338, S335, and S336 are executed. This enables the derivation of a highly accurate object detection region. Conversely, when the class data in the object detection data DET[1] and the class data in the object detection data DET[2] do not match (N in step S337), the mismatch handling process in step S339 is executed. If the data for both classes are inconsistent, it is likely that there is an error in the object detection process of either model 112[1] or 112[2]. In such cases, it is not appropriate to merge the object detection regions, and the inconsistency handling process prevents the improper merging of the object detection regions.

[0093] <<Example EX2>> Example EX2 will now be described. Example EX2 describes the modification techniques applicable to Example EX1 described above. Furthermore, the matters shown in Example EX2 are applicable to other examples described later, insofar as they do not contradict each other.

[0094] In case CS1 shown in Figures 16(a) and (b), the output class data of "car" from model B is set as the target class data, and the driver assistance device 10 performs a class conversion process on the output class data of model 112[2]. Alternatively, case CS1 may be modified so that the output class data of "automobile" from model A is set as the target class data, and the driver assistance device 10 performs a class conversion process on the output class data of model 112[1].

[0095] In case CS1 shown in Figures 16(a) and (b), if class data C[1,1] and C[2,1] are different, the same measures as when class data C[1,2] and C[2,2] are different should be taken. For example, if class data C[1,1] is "person" and class data C[2,1] is "human", then the server device 20 should generate conversion data 122 in which class data C[2,1] is set as the target class data and "person" is set as the integration class data.

[0096] In Example EX1, it is assumed that the number of models m is 2, but cases where the number of models m is 3 or more can be handled in the same way as when the number of models m is 2. For example, consider Case CS2 where "m=3" and the class data C[1,2], C[2,2], and C[3,2] for the vehicle evaluation image E[2] are "automobile", "car", and "automobile", respectively. In Case CS2, the conversion data 122_2 shown in Figure 19 is set in the server controller 21. In Case CS2, assume that the class data C[1,1], C[2,1], and C[3,1] for the person evaluation image E[1] are all "person". In the conversion data 122_2, the output class data of "car" by model B is classified as target class data. In the conversion data 122_2, the output class data of "person" by models A, B, and C, and the output class data of "automobile" by models A and C are classified as non-target class data. The conversion data 122_2 specifies that the output class data of "car" by Model B should be converted to other class data (integration class data), and that other class data represents "automobile". In case CS2, the conversion data 122_2 is stored as conversion data 122 in the memory 12 of the driver assistance device 10. Based on the conversion data 122_2 (according to the specifications in conversion data 122_2), the integration unit 113 performs a class conversion process on the output class data of Model 112[2] only if the output class data of Model 112[2] is "car". In this class conversion process, the integration unit 113 converts the output class data of Model 112[2] to "automobile" (i.e., converts it to other class data that represents "automobile"). If the class data C[3,2] is not "automobile" but, for example, "vehicle", the appropriate conversion data 122 should be generated in the same manner.

[0097] In Example EX1, it is assumed that the number of types of objects to be recognized, n, is 2. However, the number of types of objects to be recognized, n, can be any number greater than or equal to 2, and may even be "n=1". Regardless of the number of types n, the server controller 21 can set the conversion data based on the correspondence between the first to the nth class through the conversion data setting process (see Figure 13).

[0098] <<Example EX3>> Example EX3 will now be explained. Example EX3 assumes that the number of models m is 2. Also, in Example EX3, it is assumed that the number of types of objects to be recognized n for Model A is 2, while the number of types of objects to be recognized n for Model B is 3. Figure 20(a) shows an overview of Case CS3 assumed in Example EX3. In Case CS3, the first type of object (object to be recognized) is a person, and both the second type of object (object to be recognized) and the third type of object (object to be recognized) are vehicles. However, the second type of object is an automobile (passenger car) expressed as "car" in English, and the third type of object is a bus expressed as "bus" in English. Although both automobiles ("car") and buses ("bus") belong to the category of vehicles, they are distinguished in Example EX3. In Example EX3, automobiles and buses refer to different types of objects.

[0099] In Example EX3, evaluation image E[1] is a two-dimensional image containing an image of a person, which is a first type of object. In Example EX3, evaluation image E[2] is a two-dimensional image containing an image of a car (passenger car), which is a second type of object. In Example EX3, evaluation image E[3] is a two-dimensional image containing an image of a bus, which is a third type of object.

[0100] Model B has the ability to distinguish and detect objects of types 1 through 3 in object detection processing. Therefore, the number of types of objects n recognized by Model B is 3. In contrast, Model A has the ability to distinguish and detect objects of type 1, type 2, and type 3, but does not have the ability to distinguish and detect objects of type 2 and type 3. Therefore, the number of types of objects n recognized by Model A is 2. Model A treats objects of type 2 and type 3, which are the same as those of type B, as common objects. Thus, for Model MDL[1] in Server Device 20, the object to be recognized in evaluation image E[1] is a person, and the objects to be recognized in evaluation image E[2] and evaluation image E[3] are both vehicles. For Model MDL[2] in Server Device 20, the objects to be recognized in evaluation images E[1], E[2], and E[3] are a person, a car, and a bus, respectively.

[0101] In the model MDL[1] for case CS3, during object detection processing, the class data "person" is assigned to people and the class data "automobile" is assigned to vehicles. In the model MDL[2] for case CS3, during object detection processing, the class data "person" is assigned to people, the class data "car" is assigned to automobiles, and the class data "bus" is assigned to buses. Therefore, in the model set setting process for case CS3, the strings in class data C[1,1], C[1,2], and C[1,3] are set to "person", "automobile", and "automobile", respectively. In addition, in the model set setting process for case CS3, the strings in class data C[2,1], C[2,2], and C[2,3] are set to "person", "car", and "bus", respectively. As is clear from the above explanation, the concept of a vehicle includes automobiles and buses, and therefore, a vehicle is a higher-level concept of automobiles and buses, while automobiles and buses are lower-level concepts of a vehicle. In English, "automobile" is a concept that includes "car" and "bus," and therefore "automobile" is a higher-level concept than "car" and "bus," while "car" and "bus" are lower-level concepts than "automobile."

[0102] The server controller 21 identifies a first class correspondence showing the correspondence between class data C[1,1] and C[2,1], a second class correspondence showing the correspondence between class data C[1,2] and C[2,2], and a third class correspondence showing the correspondence between class data C[1,3] and C[2,3] (step S212 in Figure 13). The first class correspondence indicates that for a person, both models A and B are assigned the class data "person". The second class correspondence indicates that for an automobile, model A is assigned the class data "automobile" while model B is assigned the class data "car". In this case, based on the second class correspondence, the server controller 21 sets the conversion data so that the output class data of model B after conversion by the conversion data includes the output class data of model A (step S215 in Figure 13). The third class correspondence indicates that for a bus, model A is assigned the class data "automobile" while model B is assigned the class data "bus". In this case, the server controller 21 sets the conversion data for the bus based on the third-class correspondence so that the output class data of Model B after conversion by the conversion data includes the output class data of Model A (step S215 in Figure 13).

[0103] Specifically, in case CS3, the conversion data 122_3 shown in Figure 20(b) is set in the server controller 21, and therefore the conversion data 122_3 is stored as conversion data 122 in the memory 12 of the driver assistance device 10. The integration unit 113 may convert the output class data of model 112[1] or 112[2] to other class data, and the conversion rules are defined in the conversion data 122. As described above, in any conversion data, the output class data of each model is individually classified into target class data or non-target class data (information indicating the content of the classification is included in the conversion data). In conversion data 122_3, the output class data of "person" by model A, the output class data of "automobile" by model A, and the output class data of "person" by model B are classified into non-target class data. In conversion data 122_3, the output class data of "car" by model B and the output class data of "bus" by model B are classified into target class data.

[0104] The arbitrary conversion data is data that specifies that only the output class data classified as the target class data should be converted to other class data. The class data after this conversion is referred to as the integration class data, as already mentioned. Therefore, the conversion data 122_3 specifies that the output class data of "car" and the output class data of "bus" by Model B should be converted to other class data. Here, the other class data to which the output class data of "car" by Model B will be converted consists of two integration class data IC1 and IC2. Integration class data IC1 represents "automobile", and integration class data IC2 represents "car". That is, the class data assigned to a vehicle in Model A ("automobile") is set in integration class data IC1, and the class data assigned to an automobile in Model B ("car") is set in integration class data IC2. The other class data to which the output class data of "bus" by Model B will be converted consists of two integration class data IC3 and IC4. The integration class data IC3 represents "automobile," and the integration class data IC4 represents "bus." In other words, the class data assigned to a vehicle in Model A ("automobile") is set in integration class data IC3, and the class data assigned to a bus in Model B ("bus") is set in integration class data IC4.

[0105] In case CS3, the conversion data 122_3 is used as conversion data 122 in the actual operation process. Based on the conversion data 122_3 (according to the specifications in conversion data 122_3), the integration unit 113 performs a class conversion process on the output class data of model 112[2] only if the output class data of model 112[2] is "car" or "bus".

[0106] In the class conversion process when the output class data of model 112[2] is "car", the integration unit 113 converts the output class data of model 112[2] into other class data consisting of integration class data IC1 and IC2. After this conversion, if the output class data of model 112[1] matches either of the integration class data IC1 or IC2, the integration unit 113 considers that the output class data of model 112[1] matches the other class data. Based on this match, the integration unit 113 sets the object detection region of the inference result data by integrating the object detection region of the object detection data DET[1] and the object detection region of the object detection data DET[2].

[0107] In the class conversion process when the output class data of model 112[2] is "bus", the integration unit 113 converts the output class data of model 112[2] into other class data consisting of integration class data IC3 and IC4. After this conversion, if the output class data of model 112[1] matches either of the integration class data IC3 or IC4, the integration unit 113 considers that the output class data of model 112[1] matches the other class data. Based on this match, the integration unit 113 sets the object detection region of the inference result data by integrating the object detection region of the object detection data DET[1] and the object detection region of the object detection data DET[2].

[0108] The integration unit 113 does not perform class conversion processing on output class data that is classified as non-target class data in the conversion data 122_3. In other words, output class data classified as non-target class data is maintained as is without being modified by the integration unit 113.

[0109] Figure 21 shows a flowchart of the integration process performed by the integration unit 113. The integration process in Figure 21 consists of steps S431 to S439. The integration process in Figure 21 can be performed in step S313 of Figure 14. That is, the inference result data set in the integration process in Figure 21 may be the inference result data generated in step S313. The integration process in Figure 21 starts from step S431.

[0110] The processing content of step S431 is the same as the processing content of step S331 in Figure 17. Therefore, in step S431, the integration unit 113 determines whether the output class data of any of the models 112 contains the target class data. If the output class data of any of the models 112 contains the target class data (Y in step S431), the process proceeds to step S432. If the output class data of any of the models 112 does not contain the target class data (N in step S431), the process proceeds to step S437.

[0111] In step S432, the integration unit 113 converts the target class data to other class data based on the conversion data 122 (according to the provisions in the conversion data 122). Here, the other class data consists of multiple integration class data (two integration class data in case CS3). After step S432, the process proceeds to step S433.

[0112] In step S433, the integration unit 113 determines whether the output class data of model 112[1], which is the non-target class data, matches any of the multiple integration class data. If the output class data of model 112[1], which is the non-target class data, matches any of the multiple integration class data (Y in step S433), the integration unit 113 considers that the output class data of model 112[1] matches the other class data and proceeds to step S434. If the output class data of model 112[1] does not match any of the multiple integration class data (N in step S433), the process proceeds to step S439.

[0113] In step S434, the integration unit 113 sets the output class data of model 112[1], which is non-target class data, as the class data in the inference result data. In this case, in step S434 relating to case CS3, "automobile" is set as the class data in the inference result data. Alternatively, in step S434, the integration unit 113 may set any of the multiple integration class data as the class data in the inference result data. In this case, in step S434 relating to case CS3, "automobile", "car", or "bus" is set as the class data in the inference result data. After step S434, the process proceeds to step S435.

[0114] The processing content of step S435 is the same as the processing content of step S335 in Figure 17. Therefore, in step S435, the integration unit 113 sets the object detection region of the inference result data by integrating the object detection regions of each object detection data, and in this case, the first or second region integration method described in Example EX1 can be adopted. After step S435, the process proceeds to step S436.

[0115] The processing content of step S436 is the same as the processing content of step S336 in Figure 17. Therefore, in step S436, the integration unit 113 sets the confidence level of the inference result data based on the confidence level in each object detection data. When the processing of step S436 is performed, the integration process in Figure 21 is completed upon the execution of the processing of step S436.

[0116] The processing content of step S437 is the same as the processing content of step S337 in Figure 17. Therefore, in step S437, the integration unit 113 determines whether the output class data of each model 112 matches each other. If the output class data of each model 112 matches each other (Y in step S437), the process proceeds to step S438. If the output class data of each model 112 does not match each other (N in step S437), the process proceeds to step S439. In embodiment EX3, it is assumed that the number of models m is 2, so in step S437, the integration unit 113 determines whether the output class data of models 112[1] and 112[2] match each other. If the output class data of model 112[1] and the output class data of model 112[2] match (Y in step S437), the process proceeds to step S438. If the output class data of model 112[1] and the output class data of model 112[2] do not match (N in step S437), the process proceeds to step S439.

[0117] The processing content of step S438 is the same as the processing content of step S338 in Figure 17. Therefore, in step S438, the integration unit 113 sets the output class data of any of the models 112 as the class data in the inference result data. When proceeding to step S438, the output class data of each model 112 are the same, so regardless of which output class data is set as the class data in the inference result data, there is no difference in the class data in the inference result data. After step S438, the process proceeds to step S435.

[0118] The processing content of step S439 is the same as the processing content of step S339 in Figure 17. Therefore, in step S439, the integration unit 113 performs the mismatch handling processing described in Example EX1. When the inference result data is set in step S439, the integration processing in Figure 21 is completed. If the process proceeds to step S439, it is considered that there is an error in the object detection processing by one of the models 112. For this reason, the integration unit 113 involved in the mismatch handling processing may complete the integration processing in Figure 21 without generating inference result data (i.e., it may stop generating the inference result data). In this case, in the actual operation process in Figure 14, the process should proceed to step S311 without generating inference result data in step S313 and without performing the processing in step S314.

[0119] We will now explain the integration process in case CS3_a, assuming that the object to be recognized in the input image IN is a person. In case CS3_a, we assume that the object to be recognized was correctly detected by models 112[1] and 112[2]. In case CS3_a, the output class data of both models 112[1] and 112[2] is "person", and the output class data of "person" is not the target class data (see Figure 20(b)). Therefore, in case CS3_a, in the integration process, the output class data of model 112[1] or 112[2] is set as the class data of the inference result data, going from step S431 through step S437 to step S438. After that, the processes in steps S435 and S436 are executed.

[0120] We will now explain the integration process in Case CS3_b1, assuming that the object to be recognized in the input image IN is an automobile. In Case CS3_b1, we assume that the object to be recognized was correctly detected by Models 112[1] and 112[2]. In Case CS3_b1, the output class data of Models 112[1] and 112[2] are "automobile" and "car", respectively. The output class data of "car" from Model 112[2] is the target class data (see Figure 20(b)). Therefore, in Case CS3_b1, in the integration process, the output class data of Model 112[2] is converted to other class data consisting of integration class data IC1 ("automobile") and integration class data IC2 ("car"), proceeding from step S431 to step S432. Then, since the output class data ("automobile") of model 112[1], which corresponds to the non-target class data, matches the integration class data IC1 ("automobile"), the processing in steps S434 to S436 is executed via step S433. In step S434 of case CS3_b1, "automobile" or "car" is set as the class data in the inference result data.

[0121] We will now describe the integration process in Case CS3_b2, assuming that the object to be recognized in the input image IN is a bus. In Case CS3_b2, we assume that the object to be recognized was correctly detected by Models 112[1] and 112[2]. In Case CS3_b2, the output class data of Models 112[1] and 112[2] are "automobile" and "bus," respectively. The output class data of "bus" from Model 112[2] is the target class data (see Figure 20(b)). Therefore, in Case CS3_b2, in the integration process, the output class data of Model 112[2] is converted to other class data consisting of integration class data IC3 ("automobile") and integration class data IC4 ("bus"), proceeding from step S431 to step S432. Then, since the output class data of model 112[1] ("automobile"), which corresponds to the non-target class data, matches the integration class data IC3 ("automobile"), the processing of steps S434 to S436 is executed via step S433. In step S434 of case CS3_b2, "automobile" or "bus" is set as the class data in the inference result data.

[0122] We will now describe the integration process in case CS3_c, assuming that the object to be recognized in the input image IN is a person. However, in case CS3_c, we will assume that model 112[1] correctly detected that the object in the input image IN was a person, but model 112[2] incorrectly detected that the object in the input image IN was a car. In case CS3_c, the output class data of models 112[1] and 112[2] are "person" and "car", respectively. The output class data of "car" from model 112[2] is the target class data (see Figure 20(b)). Therefore, in case CS3_c, in the integration process, the output class data of model 112[2] is converted to other class data consisting of integration class data IC1 ("automobile") and integration class data IC2 ("car"), proceeding from step S431 to step S432. Then, the output class data ("person") of model 112[1], which corresponds to the non-target class data, differs from both the integration class data IC1 and IC2, so the process proceeds to step S439 via step S433, and the mismatch handling process in step S439 is executed. The same applies in case CS3_c when model 112[2] mistakenly detects an object in the input image IN as a bus.

[0123] We will now describe the integration process in case CS3_d, assuming that the object to be recognized in the input image IN is a person. However, in case CS3_d, model 112[2] correctly detected that the object in the input image IN was a person, but model 112[1] incorrectly detected that the object in the input image IN was a vehicle. In case CS3_d, the output class data of models 112[1] and 112[2] are "automobile" and "person," respectively, which are not the target class data (see Figure 20(b)) and do not match each other. Therefore, in case CS3_d, the integration process proceeds from step S431 through step S437 to step S439, and the mismatch handling process in step S439 is executed. Various other cases may occur, but the integration process is executed according to the flow shown in Figure 21.

[0124] Similar to Example EX1, in Example EX3, the processing content of the integration process branches depending on whether the class data in the object detection data DET[2] corresponds to the target class data. For the class data in the object detection data DET[2] to correspond to the target class data, this means that the class data in the object detection data DET[2] indicates a specific type (indicates the name of a specific type of object). In Example EX3, the name of a specific type of object is "car" or "bus". If the class data in the object detection data DET[2] indicates a specific type, the integration unit 113 converts the class data in the object detection data DET[2] to other class data according to a pre-set conversion rule (according to the conversion data 122) (S432). In Example EX3, this other class data consists of multiple integration class data. When the class data in the object detection data DET[1] corresponding to the non-target class data matches any of the multiple integration class data, the integration unit 113 considers the class data in the object detection data DET[1] to match the other class data and executes the processing in steps S434 to S436.

[0125] The processing in step S435 enables the derivation of an object detection region that accurately represents the position and size of the object to be recognized in the input image IN during inference related to object detection. When the driver assistance device 10 is configured by combining multiple existing image recognition models, different class data are often derived from each other for a common object to be recognized among the multiple image recognition models. The driver assistance device 10 enables the derivation of a highly accurate object detection region even when different class data are derived from each other for a common object to be recognized among the multiple image recognition models through integrated processing using the converted data 122.

[0126] Multiple integration class data sets, created by transforming target class data through a class conversion process, include two integration class data sets (for example, a combination of integration class data IC1 and IC2, or a combination of integration class data IC3 and IC4). Of these two integration class data sets, one (for example, "automobile") and the other (for example, "bus") have a higher-level and lower-level conceptual relationship.

[0127] In Example EX3, instead of simply converting the output class data of “car” or “bus” from Model 112[2] to the higher-level concept “automobile”, the lower-level concepts of “car” or “bus” are included in the integration class data. This makes it possible to derive inference result data that distinguishes between automobiles ("car") and buses ("bus") (S434), which is beneficial for applications that require such distinction.

[0128] <<Example EX4>> Example EX4 will now be described. Example EX4 will describe modification techniques and other methods applicable to Example EX3 described above.

[0129] In case CS3 shown in Figures 20(a) and (b), the output class data of "car" from model B is set as the target class data, and the driver assistance device 10 performs a class conversion process on the output class data of model 112[2]. Alternatively, case CS3 may be modified so that the output class data of "automobile" from model A is set as the target class data, and the driver assistance device 10 performs a class conversion process on the output class data of model 112[1]. In this case, when the output class data of model 112[1] is "automobile", the label conversion process should generate conversion data 122 to convert the output class data of model 112[1] into other class data consisting of integrated class data for "automobile", integrated class data for "car", and integrated class data for "bus".

[0130] In Example EX3, it is assumed that the number of models m is 2, but the same approach can be taken when the number of models m is 3 or more.

[0131] In Example EX3, it is assumed that the number of types of objects to be recognized, n, is 2 or 3. However, the number of types of objects to be recognized, n, in each image recognition model can be any number of 2 or more, and an image recognition model with a number of types of objects n of 1 may be mixed in among multiple image recognition models.

[0132] <<Example EX5>> Example EX5 will now be described. Example EX5 includes supplementary explanations regarding the technology of cooperation between the driver support device 10 and the server device 20.

[0133] The driver assistance device 10 has an inference processing unit that performs inference to recognize an object to be recognized in the input image IN. The inference processing unit in the driver assistance device 10 includes an inference block 114 (see Figures 3(a) and (b)) and is composed of an on-board controller 11. The driver assistance device 10 functions as an image recognition device that incorporates the inference processing unit. The server device 20 is an example of an information processing device having a setting controller, and the setting controller has a function to set conversion rules that define the conversion data 122. In the server device 20, the server controller 21 functions as the setting controller.

[0134] The server controller 21 performs unit processing consisting of steps S113 to S115 for each of the evaluation images E[1] to E[n] (Figure 11). This derives m evaluation class data (class data C[1,j] to C[m,j]) for evaluation image E[j]. The server controller 21 sets the conversion data based on the m evaluation class data obtained in each unit processing (and therefore based on at least two evaluation class data). The conversion data set by the server controller 21 is introduced into the driver support device 10 as conversion data 122. In addition, the models MDL[1] to MDL[m] used to set the conversion data are incorporated into the driver support device 10 as models 112[1] to 112[m].

[0135] In this manner, the driver assistance device 10 can set conversion data to match the class data for common recognition targets among the image recognition models (112[1]~112[m]). By using this conversion data, the object detection regions described above can be integrated, and through this integration, the accuracy of the object detection regions can be improved.

[0136] In this process, the server controller 21 sets up model sets MS[1] to MS[m] by performing unit processing on each of the evaluation images E[1] to E[n] (steps S113 to S122). Model set MS[i] has header data HD[i] and model MDL[i] (see Figure 10). Each header data has a pair of data for each evaluation image, consisting of the evaluation image and class data (evaluation class data).

[0137] By constructing a model set with this configuration within the server device 20, the server controller 21 can easily and accurately determine the relationships between class data between models MDL[1] and MDL[m], and the necessary conversion data can be easily generated.

[0138] However, the construction of the above model set is not mandatory as long as the server controller 21 can identify that the class data derived from model MDL[i] when the evaluation image E[j] is supplied to model MDL[i] is the class data C[i,j].

[0139] <<Example EX6>> Example EX6 will be described. Example EX6 will describe the modifications and other techniques for each of the techniques described above.

[0140] The integration unit 113 may generate inference result data only if the confidence levels in all object detection data DET[1] to DET[m] are above a predetermined value. If no inference result data is generated for a particular object to be recognized in the input image IN, the inference result processing related to that object (step S314 in Figure 14) is not executed. The confidence level does not need to be included in the inference result data.

[0141] For the sake of concretization and simplification of the explanation, in each of the embodiments described above, it was assumed that only one object to be recognized is detected by each model 112 from a single input image IN. However, in reality, multiple objects to be recognized are often detected by each model 112 from a single input image IN. For example, consider the case where a first and a second object to be recognized are detected by models 112[1] and 112[2] from a single input image IN. In this case, two object detection data DET[1] for the first and second objects to be recognized are output from model 112[1], and two object detection data DET[2] for the first and second objects to be recognized are output from model 112[2]. At this time, the integration unit 113 can identify corresponding pairs of object detection regions based on the IoU between each object detection region of the two object detection data DET[1] and each object detection region of the two object detection data DET[2], and perform integration processing for each identified pair.

[0142] While passenger cars or trucks traveling on public roads are primarily envisioned as vehicle V1, the type of vehicle V1 is arbitrary. Vehicle V1 may also be an industrial vehicle such as a forklift, or an agricultural vehicle such as a tractor. The driving assistance device 10 may be mounted on any type of mobile body. Vehicle V1 is a type of mobile body. Examples of mobile bodies other than vehicle V1 include airplanes, ships, trains, and mobile robots. User U1 is the occupant of the mobile body and the driver who operates the mobile body in order to move it.

[0143] A program that causes a computer device to execute any method described in the embodiments of the present invention, and a non-volatile recording medium on which such program is recorded, are included within the scope of the embodiments of the present invention. The program that causes a computer (computer device) to execute any method described in the embodiments of the present invention may be a subprogram incorporated into any main program or called by any main program. The image recognition device according to the present invention, exemplified as a driving assistance device 10, includes a computer capable of executing any program. The in-vehicle controller 11 itself or the arithmetic processing unit provided in the in-vehicle controller 11 may be considered to be the computer. The information processing device according to the present invention, exemplified as a server device 20, includes a computer capable of executing any program. The server controller 21 itself or the arithmetic processing unit provided in the server controller 21 may be considered to be the computer. Any processing in the embodiments of the present invention may be realized by hardware such as a semiconductor integrated circuit, software corresponding to the above program, or a combination of hardware and software.

[0144] The image recognition device according to the present invention is not limited to applications of moving objects such as vehicles V1. In other words, it is not essential that the image recognition device according to the present invention be mounted on a moving object, and the user U1 does not have to be an occupant of the moving object. The image recognition device according to the present invention can be applied to any application.

[0145] The embodiments of the present invention can be modified in various ways as appropriate within the scope of the technical idea set forth in the claims. The embodiments described above are merely examples of embodiments of the present invention, and the meaning of the terms of the present invention or each constituent element is not limited to those described above. The specific numerical values ​​shown in the above description are merely examples and can, of course, be changed to various numerical values. [Explanation of Symbols]

[0146] V1 Vehicle U1 User CM Camera 10. Driving support systems 11. In-vehicle controller 111 Image acquisition unit 112 Image Recognition Models 113 Integration Department 114 Inference Blocks 12 memory 121 Model Data 122 Converted data 13 Communications Department 14 HMI 141 Microphone 142 speakers 143 Display device 20 Server Devices 21 Server Controllers 22 memory 23 Communications Department MS[1]~MS[m] Model Set HD[1]~HD[m] Header Data MDL[1]~MDL[m] Pre-trained AI models E[1]~E[n] Evaluation images C[1,1]~C[m,n] Class Data

Claims

1. An image recognition device having a controller that performs inference for recognizing target objects in an image, The controller has a first image recognition model and a second image recognition model, and performs object detection processing using the first image recognition model and the second image recognition model respectively to detect regions in the input image where the object to be recognized is determined to exist as object detection regions, thereby generating first object detection data with the first image recognition model and second object detection data with the second image recognition model, and generating inference result data based on at least one of the first object detection data and the second object detection data. The first object detection data, the second object detection data, and the inference result data each include class data indicating the type of object within the object detection region and positional shape data of the object detection region. The controller, when the class data in the second object detection data indicates a specific type, converts the class data in the second object detection data to other class data according to a preset conversion rule, and when the class data in the first object detection data matches the other class data, sets the class data in the first object detection data or the other class data as the class data in the inference result data, and generates the position shape data in the inference result data by integrating the position shape data in the first object detection data and the position shape data in the second object detection data. Image recognition device.

2. The aforementioned other class data has multiple integration class data, The controller, when the class data in the second object detection data indicates the specific type, converts the class data in the second object detection data into the plurality of integration class data according to the conversion rule. When the class data in the first object detection data matches any of the plurality of integration class data, the controller considers that the class data in the first object detection data matches the other class data, sets either the class data in the first object detection data or the plurality of integration class data as the class data in the inference result data, and generates the position shape data in the inference result data by integrating the position shape data in the first object detection data with the position shape data in the second object detection data. The image recognition device according to claim 1.

3. The plurality of integration class data include first integration class data and second integration class data, The object types indicated in the first integration class data and the object types indicated in the second integration class data are in a relationship of higher-level and lower-level concepts. The image recognition device according to claim 2.

4. The controller, when the class data in the second object detection data indicates the specific type, converts the class data in the second object detection data to the other class data according to the conversion rule, and when the class data in the first object detection data and the other class data do not match, sets the class data and position shape data in the first object detection data to the class data and position shape data in the inference result data, sets the class data and position shape data in the second object detection data to the class data and position shape data in the inference result data, or stops generating the inference result data. or an image recognition device according to any one of claims 1 to 3.

5. If the class data in the second object detection data indicates a different type from the specified type, the controller compares the class data in the first object detection data with the class data in the second object detection data. When the class data in the first object detection data matches the class data in the second object detection data, the class data in the first object detection data or the class data in the second object detection data is set as the class data in the inference result data, and the position shape data in the inference result data is generated by integrating the position shape data in the first object detection data and the position shape data in the second object detection data. If the class data in the first object detection data and the class data in the second object detection data do not match, the system may set the class data and position / shape data in the first object detection data to the class data and position / shape data in the inference result data, set the class data and position / shape data in the second object detection data to the class data and position / shape data in the inference result data, or stop generating the inference result data. or an image recognition device according to any one of claims 1 to 3.

6. An image recognition device according to any one of claims 1 to 3, An image recognition system comprising an information processing device having a setting controller for setting the conversion rules, The setting controller has a first model and a second model capable of performing the object detection process, and performs unit processing on each of the first to nth evaluation images, which include images of different types of objects. The setting controller, in unit processing for the j-th evaluation image, inputs the j-th evaluation image to the first model and causes the first model to perform the object detection process, thereby causing the first model to derive first evaluation class data indicating the type of object in the j-th evaluation image; inputs the j-th evaluation image to the second model and causes the second model to perform the object detection process, thereby causing the second model to derive second evaluation class data indicating the type of object in the j-th evaluation image; n represents an integer greater than or equal to 2, and j represents a natural number less than or equal to n. The setting controller sets conversion data indicating the conversion rule based on the first evaluation class data and the second evaluation class data obtained in each unit processing, The controller in the image recognition device has the conversion data and has the first model and the second model as the first image recognition model and the second image recognition model. Image recognition system.

7. The setting controller sets a first model set having the first model and first header data and a second model set having the second model and second header data by performing the unit processing for each of the first to nth evaluation images. The first header data has a pair of the evaluation image and the first evaluation class data for each of the evaluation images. The second header data has a pair of the evaluation image and the second evaluation class data for each evaluation image. The image recognition system according to claim 6.

8. An image recognition method for performing inference to recognize an object to be recognized in an image, comprising an inference step performed by an image recognition device having a first image recognition model and a second image recognition model, In the inference step, an object detection process is performed using the first image recognition model and the second image recognition model respectively to detect regions in the input image where the object to be recognized is determined to exist as object detection regions, thereby generating first object detection data with the first image recognition model and second object detection data with the second image recognition model, and generating inference result data based on at least one of the first object detection data and the second object detection data. The first object detection data, the second object detection data, and the inference result data each include class data indicating the type of object within the object detection region and positional shape data of the object detection region. In the inference step, if the class data in the second object detection data indicates a specific type, the class data in the second object detection data is converted to other class data according to a pre-set conversion rule. When the class data in the first object detection data matches the other class data, the class data in the first object detection data or the other class data is set as the class data in the inference result data. The position shape data in the inference result data is then generated by integrating the position shape data in the first object detection data with the position shape data in the second object detection data. Image recognition methods.

9. The information processing device having a first model and a second model capable of performing the object detection process includes a setting step for setting the conversion rule, In the setting step, unit processing is performed on each of the first to nth evaluation images, which include images of objects of different types from each other. In the unit processing for the j-th evaluation image in the setting step, the j-th evaluation image is input to the first model and the first model is made to perform the object detection process, thereby causing the first model to derive first evaluation class data indicating the type of object in the j-th evaluation image, and the j-th evaluation image is input to the second model and the second model is made to perform the object detection process, thereby causing the second model to derive second evaluation class data indicating the type of object in the j-th evaluation image. n represents an integer greater than or equal to 2, and j represents a natural number less than or equal to n. In the setting step, conversion data indicating the conversion rule is set based on the first evaluation class data and the second evaluation class data obtained in each unit processing. In the inference step, the class data in the second object detection data is converted to the other class data based on the converted data. The first model and the second model are incorporated into the image recognition device as the first image recognition model and the second image recognition model. The image recognition method according to claim 8.

10. A method for setting conversion data executed in an information processing device, comprising a first step and a second step, In the aforementioned information processing device, a first model and a second model capable of performing object detection processing are formed. In the first step, using the first model and the second model, unit processing is performed on each of the first to nth evaluation images, which include images of objects of different types. In the unit processing for the j-th evaluation image, the j-th evaluation image is input to the first model and the first model is made to perform the object detection process, thereby outputting first evaluation class data indicating the type of object in the j-th evaluation image from the first model, and the j-th evaluation image is input to the second model and the second model is made to perform the object detection process, thereby outputting second evaluation class data indicating the type of object in the j-th evaluation image from the second model, n represents an integer greater than or equal to 2, and j represents a natural number less than or equal to n. In the second step, conversion data is set according to the relationship between the first evaluation class data and the second evaluation class data obtained in each unit processing. How to set up the conversion data.

Images