Information processing apparatus and information processing method

Abstract

The present disclosure relates to an information processing device and an information processing method capable of further improving recognition performance. An information processing device on a server side has a predetermined number of recognition units each of which sets a model that performs image recognition on a predetermined number of vehicles and is updated by performing unsupervised learning, a recognition unit that performs image recognition on an image on which image recognition has been performed on the predetermined number of vehicles, and an evaluation value calculation unit that evaluates recognition results from the predetermined number of recognition units and calculates an evaluation value for each recognition unit. An information processing device on a vehicle side has an execution unit that performs unsupervised learning, and a determination unit that determines whether or not learning is correctly performed for a model updated in the execution unit based on the evaluation value obtained on the server side. The present technology can be applied to, for example, a vehicle for automated driving.

Description

Technical Field

[0001] This disclosure relates to information processing apparatus and information processing methods, and more specifically, to information processing apparatus and information processing methods that can further improve recognition performance. Background Technology

[0002] Typically, learning using supervised data is performed in image recognition using, for example, convolutional neural networks (CNNs) in deep learning.

[0003] Furthermore, for example, Patent Document 1 discloses a neural network system in which a management device receives weight parameters obtained through learning in multiple terminal devices and selects the optimal weight parameters.

[0004] Reference List

[0005] Patent documents

[0006] Patent Document 1: Japanese Patent Application Publication No. 2017-174298 Summary of the Invention

[0007] The problem to be solved by the present invention

[0008] Incidentally, in performing conventional image recognition learning, not only is there the cost of preparing supervised data, but the recognition performance of the data will also decrease when the input data has a distribution different from that of the supervised data.

[0009] This disclosure is made in view of this situation and is intended to further improve recognition performance.

[0010] Problem Solution

[0011] The information processing apparatus according to the first aspect of this disclosure includes: a predetermined number of recognition units, each recognition unit having a model updated by performing image recognition and unsupervised learning in a predetermined number of terminals, the recognition units being configured to perform image recognition on images that have already undergone image recognition in the predetermined number of terminals; and an evaluation value calculation unit, the evaluation value calculation unit being configured to evaluate the recognition results in the predetermined number of recognition units and calculate an evaluation value for each recognition unit.

[0012] The information processing method according to the first aspect of this disclosure includes: performing image recognition on an image that has been image recognized in a predetermined number of terminals using an information processing device; setting a model updated by performing image recognition and unsupervised learning in a predetermined number of terminals for each recognition unit; and evaluating the recognition results in the predetermined number of recognition units and calculating an evaluation value for each recognition unit.

[0013] In a first aspect of this disclosure, image recognition is performed on an image that has been image recognized in a predetermined number of terminals in a predetermined number of recognition units; a model is set for each recognition unit by performing image recognition and performing unsupervised learning in a predetermined number of terminals; and the recognition results in the predetermined number of recognition units are evaluated, and an evaluation value is calculated for each recognition unit.

[0014] The information processing apparatus according to a second aspect of this disclosure includes: an execution unit configured to perform unsupervised learning on a model of a recognizer that performs image recognition on an image; and a determination unit that determines whether the learning has been correctly performed on a model updated by learning in the execution unit based on an evaluation value obtained by performing image recognition on a predetermined number of models that are different from each other, wherein if the determination unit determines that the learning has not been correctly performed, the execution unit returns the model to the model before the learning was performed.

[0015] The information processing method according to the second aspect of this disclosure includes: performing unsupervised learning on a model of a recognizer that performs image recognition of an image using an information processing device; and determining whether the learning has been correctly performed on the model updated by the learning based on an evaluation value obtained by performing image recognition on a predetermined number of models that are different from each other, wherein if it is determined that the learning has not been correctly performed, the model is returned to the model before the learning was performed.

[0016] In a second aspect of this disclosure, unsupervised learning is performed on a model of a recognizer that performs image recognition on an image; and the model updated by the learning is determined to have been correctly learned based on an evaluation value obtained by performing image recognition on a predetermined number of models that are different from each other, wherein if it is determined that the learning was not correctly performed, the model is reverted to the model before the learning was performed. Attached Figure Description

[0017] Figure 1 This is a diagram used to explain conventional learning processes.

[0018] Figure 2 This is a diagram used to explain the learning process applied to this technology.

[0019] Figure 3 This is a diagram used to explain the learning process applied to this technology.

[0020] Figure 4 This is a block diagram illustrating a configuration example of an embodiment of a learning system to which this technology is applied.

[0021] Figure 5 This is a block diagram illustrating an example configuration of the information processing equipment on the vehicle side.

[0022] Figure 6 This is a block diagram illustrating a configuration example of a domain adaptive processing execution unit.

[0023] Figure 7 This is a block diagram illustrating a configuration example of an information processing device on the server side.

[0024] Figure 8 It is a flowchart used to explain the information processing to be performed on the vehicle side.

[0025] Figure 9 This is a flowchart used to interpret the adaptive processing of the domain.

[0026] Figure 10 It is a flowchart used to explain the information processing to be performed on the server side.

[0027] Figure 11 This is a block diagram illustrating an example configuration of a computer to which the present technology is applied.

[0028] Figure 12 This is a block diagram illustrating an example of a schematic configuration of a vehicle control system.

[0029] Figure 13 This is an explanatory diagram illustrating an example of the installation location of the vehicle external information detection unit and the image acquisition unit. Detailed Implementation

[0030] In the following text, specific embodiments of the application of this technology will be described in detail with reference to the accompanying drawings.

[0031] <Learning Processing>

[0032] First, refer to Figures 1 to 3 Describe the learning process.

[0033] Figure 1 It is a schematic diagram illustrating the processing flow in a conventional learning process.

[0034] like Figure 1 As shown, in the conventional learning process, a Recognizer CNN that performs image recognition using deep learning is used, and labeled images with pre-created labels for classifying multiple objects included in the image are input into the Recognizer CNN.

[0035] For example, the CNN recognizer performs image recognition on a labeled image, identifying multiple objects included in the labeled image and outputting the recognition results obtained by classifying the corresponding objects. Then, the recognition results output from the CNN recognizer are compared with the correct answer labels used to label the image, and feedback is performed to the CNN recognizer to make the recognition results closer to the correct answer labels.

[0036] As mentioned above, in conventional learning processes, correct answer labels are used to perform learning that enables the CNN recognizer to perform more accurate image recognition.

[0037] Therefore, we attempt to achieve accurate image recognition without using correct answer labels, as in the reference. Figure 2 and 3 As stated above.

[0038] Figure 2 and 3 This is a schematic diagram illustrating the processing flow in the learning process for which this technology is applied. Although two types of images will first be illustrated below—images with correct answer labels (hereinafter referred to as labeled images) and images without correct answer labels (hereinafter referred to as unlabeled images)—to facilitate understanding of the description, correct answer labels are unnecessary in the learning process for which this technology is applied, as... Figure 2 and 3 As shown in the diagram. That is, no correct answer labels are prepared for the two types of input images used in the learning process to which this technique is applied, and the learning process is performed using such images.

[0039] like Figure 2 and 3 As shown, in the learning process to which this technique is applied, two recognizers, CNN1 and CNN2, are used, and there may be no correct answer label corresponding to the image input to recognizer CNN1.

[0040] For example, in Figure 2 In the learning process shown, the recognizer CNN1 performs image recognition on the unlabeled image and outputs a recognition result (unlabeled) obtained by classifying multiple objects included in the unlabeled image. Moreover, this recognition result is input into the recognizer CNN2, and learning is performed using the correct answer value (labeled / unlabeled) to enable the recognizer CNN2 to distinguish between the recognition result (labeled) and the recognition result (unlabeled).

[0041] Then, by feeding the learning results from CNN2 back to CNN1, CNN1 can make its recognition results for unlabeled images closer to those for labeled images. Therefore, in Figure 2In the example shown, relative to the boundary between the road and the sidewalk on the right side of the recognition result, the CNN1 recognizer can accurately classify the parts of the object in the unlabeled image that were not accurately classified.

[0042] In addition, Figure 3 The learning process shown uses two images captured in different environments (an image from the first environment and an image from the second environment).

[0043] That is, the recognizer CNN1 performs image recognition on the image of the second environment and outputs the recognition result (second environment) obtained by classifying the multiple objects included in the image of the second environment. Moreover, this recognition result is input into the recognizer CNN2, and the correct answer value (first environment / second environment) is used to perform learning that enables the recognizer CNN2 to distinguish between the recognition result (first environment) and the recognition result (second environment).

[0044] Then, by feeding the learning results from CNN2 back to CNN1, CNN1's recognition results for the second environment are made closer to those for the first environment. Therefore, CNN1 can accurately classify the parts of the second environment image where the illuminated objects have not yet been accurately classified, similar to... Figure 2 The example shown illustrates this. Therefore, learning processes for accurate image recognition are performed using images of a second environment for which no correct answer label exists, and images of the first environment.

[0045] As described above, in the learning process to which this technology is applied, the CNN1 recognizer can accurately perform image recognition without preparing correct answer labels, and can perform accurate image recognition on unlabeled images, that is, the recognition performance can be improved.

[0046] Note that, as referenced Figure 2 and 3 The learning process described, performed in an unsupervised manner using images for which labels are not necessarily prepared to provide the correct answers, will be referred to below as domain adaptation processing.

[0047] <Configuration Example of a Learning System>

[0048] Figure 4 This is a block diagram illustrating a configuration example of an embodiment of a learning system to which this technology is applied.

[0049] Figure 4The learning system 11 shown is configured by connecting N vehicles 13-1 to 13-N to server 14 via network 12, and each of vehicles 13-1 to 13-N includes a recognizer. Note that vehicles 13-1 to 13-N are similarly configured and, where there is no need to distinguish them, are referred to as vehicle 13 below.

[0050] In learning system 11, domain adaptive processing is learned in real time (e.g., at the frame rate level) for images captured in vehicles 13-1 to 13-N within each of vehicles 13-1 to 13-N. Then, in learning system 11, the N images captured in vehicles 13-1 to 13-N and the models of the N recognizers are uploaded to server 14 via network 12.

[0051] Therefore, in the learning system 11, N images accumulated from vehicles 13-1 to 13-N in the server 14 are input to all N recognizers configured with corresponding models for vehicles 13-1 to 13-N. Then, in the server 14, a comprehensive recognition result is calculated, in which the recognition results obtained by the N recognizers are combined, and each recognition result is evaluated using the comprehensive recognition result and the corresponding recognition results of the N recognizers. For example, an evaluation value for evaluating the recognition result is calculated for each constant time period longer than the frame rate at which the images are captured in vehicle 13.

[0052] Subsequently, in the learning system 11, if the evaluation value representing the assessment of the recognition result is equal to or less than a certain value, feedback is executed from the server 14 to the recognizer vehicle 13, which includes the model from which the recognition result has been obtained. In response, in the vehicle 13, the model at this point is rolled back to its state before learning, and the input data used for learning is discarded.

[0053] Here, in the learning system 11, the recognizers included in vehicles 13-1 to 13-N are similarly configured. Images captured in the respective vehicles 13 are input to the recognizers, and the recognition results obtained by performing image recognition on the images are output. Furthermore, in the server 14, evaluation and feedback are not always performed, but only when verifying domain adaptive processing in vehicles 13-1 to 13-N, for example, approximately once every few hours. That is, in the learning system 11, learning processing is performed at a high frequency in vehicles 13-1 to 13-N, while evaluation and feedback are performed at a low frequency in the server 14.

[0054] Figure 5 This is a block diagram illustrating an example configuration of the information processing device 21 included in vehicle 13.

[0055] like Figure 5As shown, the information processing device 21 includes an image acquisition unit 31, a storage unit 32, a domain adaptive processing execution unit 33, a sending unit 34, a receiving unit 35, a determination unit 36, and a recognition processing unit 37.

[0056] The image acquisition unit 31 acquires an image captured by an image picking device (not shown) installed on the vehicle 13 and supplies the image to the domain adaptive processing execution unit 33 and the recognition processing unit 37.

[0057] Storage unit 32 stores the model of the recognizer that performs image recognition in information processing device 21. For example, storage unit 32 stores the initial model as the model set to the initial state, the latest model after being updated in domain adaptive processing execution unit 33, the model immediately before the update, etc.

[0058] The domain adaptive processing execution unit 33 sets the model loaded from the storage unit 32 into the recognizer and performs domain adaptive processing on the image supplied from the image acquisition unit 31. Then, the domain adaptive processing execution unit 33, for example, performs evaluation and feedback timing in the server 14, supplies the model set in the recognizer and the image for image recognition to the sending unit 34. Furthermore, if the recognizer's model has been correctly learned, the domain adaptive processing execution unit 33 can supply the model to the storage unit 32 to update the model. Note that, as will be discussed later... Figure 6 Describe the detailed configuration of the domain adaptive processing execution unit 33.

[0059] The sending unit 34 acquires additional information (e.g., vehicle 13 location information, vehicle type information, image acquisition equipment information, image acquisition condition information, etc.) from the high-level control device (not shown) along with the image and model supplied by the domain adaptive processing execution unit 33, and transmits it via... Figure 1 Network 12 sends additional information to server 14.

[0060] The receiving unit 35 receives the evaluation value sent from the server 14 via the network 12 and supplies the evaluation value to the judgment unit 36.

[0061] The determination unit 36 ​​compares the evaluation value obtained in the server 14 with the predetermined evaluation value threshold, determines whether the learning has been correctly performed in the domain adaptive processing execution unit 33, and notifies the domain adaptive processing execution unit 33 of the determination result.

[0062] The recognition processing unit 37 reads the latest model from the storage unit 32, performs recognition processing on the image supplied by the image acquisition unit 31 using the recognizer for which the model is set, and outputs the recognition result obtained through the recognition processing. For example, the recognition result obtained by the recognition processing unit 37 is input into the autonomous driving system of the vehicle 13, and the recognition result is used to control the autonomous driving of the vehicle 13. Therefore, when the recognition processing unit 37 uses the model correctly learned in the domain adaptive processing execution unit 33, the recognition performance of the recognition processing unit 37 is improved, and safer autonomous driving can be achieved.

[0063] Figure 6 It is a diagram. Figure 5 A block diagram illustrating a configuration example of the domain adaptive processing execution unit 33.

[0064] like Figure 6 As shown, the domain adaptive processing execution unit 33 includes an environmental feature quantity calculation unit 41, a domain determination unit 42, and an update value calculation unit 43, and the recognizer is configured with, for example, the environmental feature quantity calculation unit 41 and the domain determination unit 42.

[0065] Environmental characteristic calculation unit 41 uses predetermined coefficients based on... Figure 5 The image acquisition unit 31 provides environmental feature quantities that indicate the characteristics of each pixel in the image and supplies these environmental feature quantities to the domain determination unit 42.

[0066] The domain determination unit 42 classifies the environmental features supplied from the environmental feature quantity calculation unit 41 according to a predetermined threshold, and performs domain determination to determine a region including pixels indicating environmental feature quantities classified by the threshold as a domain. Then, the domain determination unit 42 will determine the region as a domain by comparing the images (e.g., as shown in the image). Figure 2 and 3 The identification results obtained by classifying the domain determination results of each irradiated object included in the diagram are supplied to the update value calculation unit 43.

[0067] The update value calculation unit 43 calculates updated values ​​for environmental characteristic quantities to update the coefficients used in the environmental characteristic quantity calculation unit 41 to calculate the environmental characteristic quantities, and calculates updated values ​​for domain determination based on the domain determination results supplied from the domain determination unit 42 to update the threshold used in the domain determination unit 42 to perform domain determination. Then, the update value calculation unit 43 feeds back the updated values ​​for environmental characteristic quantities to the environmental characteristic quantity calculation unit 41, and feeds back the updated values ​​for domain determination to the domain determination unit 42.

[0068] Furthermore, the update value calculation unit 43 supplies the coefficients used to calculate the environmental feature quantities in the environmental feature quantity calculation unit 41 and the threshold used to perform domain determination in the domain determination unit 42 as the model of the recognizer to the system. Figure 5 The sending unit 34 in the middle.

[0069] Figure 7 This is a block diagram illustrating a configuration example of the information processing device 22 included in server 14.

[0070] like Figure 7 As shown, the information processing device 22 includes a receiving unit 51, N identification units 52-1 to 52-N, a synthesis unit 53, an evaluation and calculation unit 54, and a sending unit 55.

[0071] The receiving unit 51 receives images, models, and additional information transmitted from N vehicles 13-1 to 13-N. Then, the receiving unit 51 supplies the models of the recognizers from the N vehicles 13-1 to 13-N to the corresponding recognition units 52-1 to 52-N. Subsequently, the receiving unit 51 supplies N pieces of additional information and each of the N images to all recognition units 52-1 to 52-N.

[0072] Recognition units 52-1 to 52-N each set up N vehicle models 13-1 to 13-N. Then, recognition units 52-1 to 52-N input images obtained by weighting the models with additional information, and supply the recognition results obtained by performing image recognition to integration unit 53 and evaluation calculation unit 54.

[0073] The integration unit 53 aggregates the N recognition results supplied from the recognition units 52-1 to 52-N, calculates the integrated recognition result by, for example, their weighted average, and supplies the integrated recognition result to the evaluation calculation unit 54.

[0074] The evaluation calculation unit 54 uses the integrated identification results supplied from the integration unit 53 and the identification results supplied from the identification units 52-1 to 52-N to calculate the evaluation value for individually evaluating the model set for the corresponding identification units 52-1 to 52-N, and supplies the evaluation value to the sending unit 55.

[0075] The sending unit 55 will send the evaluation values ​​of the models of each recognizer supplied by the evaluation calculation unit 54 to the corresponding vehicles 13 in the N vehicles 13-1 to 13-N respectively.

[0076] Here, the evaluation of the model by the synthesis unit 53 and the evaluation calculation unit 54 will be described. For example, thresholding is performed using the average load value based on distance.

[0077] That is, the synthesis unit 53 and the evaluation calculation unit 54 can use the pixel value I of the image recognition result to be evaluated. n,ij The pixel value R of the image recognition result to be referenced ijThe index set N of vehicle 13, the index set W of image width, the index set H of image height, and the inter-domain distance d calculated using additional information. n The following expression (1) is used to calculate the comprehensive recognition result R. ij and evaluation value D n .

[0078] [Expression 1]

[0079]

[0080] In addition, the inter-domain distance d n It can be calculated based on a rule base, or by using the distance from the center after clustering with additional information.

[0081] For example, this can be achieved by using a constant a (>0), the direct distance d from the starting point, and a predefined difference b for each vehicle model. car The following expression (2) is used to calculate the inter-domain distance d based on the rule base. n Here, differential b can be comprehensively configured based on factors such as the vehicle's height, the attachment position and orientation of the image acquisition device, and the type of image acquisition device (resolution and viewing angle). car .

[0082] [Expression 2]

[0083] d n =a·d+b car ···(2)

[0084] In addition, a constant a (>0), the number of boundaries k, and a predefined difference b for each vehicle model can be used. car The following expression (3) is used to calculate the inter-domain distance d based on the rule base. n Here, for example, the boundaries in the number of boundaries k can be unique classifications such as cities, counties, or states that cross boundaries.

[0085] [Expression 3]

[0086] d n =a·k+b car ···(3)

[0087] Note that the right-hand side of expressions (2) and (3) depends on n, but this is omitted for the sake of simplicity.

[0088] Furthermore, by pre-clustering various previously acquired vehicle 13, environment, and other additional information using methods such as k-means and calculating the cluster centers (centroids), the distance from the center after clustering with additional information is obtained. Then, using the additional information a corresponding to the input image... iand the cluster center c to which the additional information corresponding to each identifier belongs. n Perform the operation on the following expression (4).

[0089] [Expression 4]

[0090] d n =||a i -c n || ···(4)

[0091] Then, the inter-domain distance d obtained in this way n Used in the above expression (1), the comprehensive recognition result R is calculated. ij and evaluation value D n .

[0092] In addition, the evaluation value D n The evaluation value D is sent from server 14 to each of vehicles 13-1 to 13-N, and the determination unit 36 ​​evaluates the value as described above. n With the evaluation value threshold D thresh，n Compare them. Then, if the evaluation value D n Equal to or greater than the evaluation value threshold D thresh，n If so, then the learning process has been performed correctly.

[0093] Information Processing

[0094] Figure 8 This is a flowchart used to explain the information processing on the vehicle 13 side to be performed by the information processing device 21.

[0095] For example, when vehicle 13 starts driving and the image capture device (not shown) included in vehicle 13 starts capturing images, processing begins, and the domain adaptive processing execution unit 33 loads the initial model from storage unit 32 and sets the initial model in the recognizer in step S11.

[0096] In step S12, the image acquisition unit 31 acquires the image captured by the image picking device and supplies the image to the domain adaptive processing execution unit 33.

[0097] In step S13, the domain adaptive processing execution unit 33 performs domain adaptive processing on the image supplied by the image acquisition unit 31 in step S12 (see [link]). Figure 9 (Flowchart in the diagram), and update the model set in the recognizer.

[0098] In step S14, when the time is reached to perform evaluation and feedback in server 14, domain adaptive processing execution unit 33 supplies the image and model to sending unit 34, and sending unit 34 sends the image, model and additional information to server 14.

[0099] In step S15, the receiving unit 35 receives information processing performed in the server 14 (see [link]). Figure 10 The evaluation value D is sent as a result of the flowchart in the diagram. n The evaluation value is then supplied to the decision unit 36.

[0100] In step S16, the determination unit 36 ​​will use the evaluation value D supplied from the receiving unit 35 in step S15. n With the evaluation value threshold D thresh,n The comparison is performed, and it is determined whether the learning was correctly executed in the domain adaptive processing execution unit 33. For example, if the evaluation value D... n Equal to or greater than the evaluation value threshold D thresh,n (D n ≥D thresh,n If the evaluation value D is positive, then decision unit 36 ​​determines that the learning has been performed correctly, or if the evaluation value D is negative. n Less than the evaluation threshold D thresh,n (D n <D thresh,n If the result is negative, then the learning process has not been performed correctly.

[0101] If the determination unit 36 ​​determines in step S16 that the learning has not been correctly executed in the domain adaptive processing execution unit 33, the process proceeds to step S17.

[0102] In step S17, the determination unit 36 ​​notifies the domain adaptive processing execution unit 33 of the determination result, and the domain adaptive processing execution unit 33 restores the model set in the recognizer to the original model before the update. Afterwards, the process returns to step S12, and similar processing is repeated thereafter.

[0103] On the other hand, if the determination unit 36 ​​determines in step S16 that the learning has been correctly performed in the domain adaptive processing execution unit 33, the process proceeds to step S18.

[0104] In step S18, the domain adaptive processing execution unit 33 determines whether to continue updating the information processing of the recognizer's model.

[0105] If the domain adaptive processing execution unit 33 determines in step S18 to continue updating the information processing of the recognizer's model, the processing returns to step S12 and similar processing is repeated thereafter. On the other hand, if the domain adaptive processing execution unit 33 determines in step S18 not to continue updating the information processing of the recognizer's model, the processing terminates.

[0106] As described above, the information processing device 21 can determine whether the model of the recognizer, which has been updated through domain adaptation processing, has been correctly learned based on the evaluation value from the server 14. Therefore, compared to determining whether the information processing device 21 has correctly performed the learning process alone, a more accurate determination can be made, and recognition performance can be improved.

[0107] Figure 9 It is used to explain in Figure 8 The flowchart shows the domain adaptation process to be performed in step S13.

[0108] In step S21, the domain adaptive processing execution unit 33 determines whether an image has been acquired from the image acquisition unit 31, and terminates the processing if it determines that an image has not been acquired. Then, if it is determined in step S21 that an image has been acquired, the processing proceeds to step S22.

[0109] In step S22, the environmental feature calculation unit 41 calculates environmental feature quantities indicating the features of each pixel from the acquired image using predetermined coefficients, and supplies the environmental feature quantities to the domain determination unit 42.

[0110] In step S23, the domain determination unit 42 classifies the environmental feature quantities supplied from the environmental feature quantity calculation unit 41 in step S22 according to a predetermined threshold, performs domain determination to determine a region including pixels indicating environmental feature quantities classified by the threshold as a domain, and supplies the domain determination result obtained as a result to the update value calculation unit 43.

[0111] In step S24, based on the domain determination result supplied from the domain determination unit 42 in step S23, the update value calculation unit 43 calculates the environmental feature quantity calculation update value for updating the coefficients used to calculate the environmental feature quantity in the environmental feature quantity calculation unit 41, and the domain determination update value for updating the threshold used to determine the domain in the domain determination unit 42.

[0112] In step S25, the update value calculation unit 43 feeds back the updated environmental feature value to the environmental feature value calculation unit 41, and feeds back the updated domain determination value to the domain determination unit 42. Therefore, the process terminates after the model of the recognizer configured with the environmental feature value calculation unit 41 and the domain determination unit 42 is updated.

[0113] Figure 10 This is a flowchart used to explain the information processing to be performed by the information processing device 22 on the server 14 side.

[0114] In step S31, the receiving unit 51 receives the data from the receiving unit. Figure 8In step S14, images, models, and additional information are sent by N vehicles 13-1 to 13-N. Then, receiving unit 51 supplies the models of the recognizers in the N vehicles 13-1 to 13-N to the corresponding recognition units 52-1 to 52-N respectively, and supplies N images and N pieces of additional information to all recognition units 52-1 to 52-N respectively.

[0115] In step S32, recognition units 52-1 to 52-N respectively set up N vehicle models 13-1 to 13-N and perform image recognition on the images weighted with additional information. Then, recognition units 52-1 to 52-N supply the recognition results obtained by performing image recognition to integration unit 53 and evaluation calculation unit 54.

[0116] In step S33, the integration unit 53 calculates the integrated recognition result R from the N recognition results supplied by the recognition units 52-1 to 52-N in step S32. ij The comprehensive identification results are then supplied to the evaluation calculation unit 54.

[0117] In step S34, the evaluation calculation unit 54 uses the comprehensive identification result R supplied from the synthesis unit 53 in step S33. ij Based on the recognition results supplied from recognition units 52-1 to 52-N in step S32, an evaluation value D is calculated for individually evaluating the model set up for the respective recognition units 52-1 to 52-N. n Then, the evaluation calculation unit 54 will evaluate the value D. n Supply to sending unit 55.

[0118] In step S35, the sending unit 55 sends the evaluation value D supplied from the evaluation calculation unit 54 in step S34. n The evaluation value D is sent to the corresponding vehicle 13-n among N vehicles 13-1 to 13-N. Then, the evaluation value D is sent to the sending unit 55. n After sending the data to the corresponding N vehicles 13-1 to 13-N, the processing terminates.

[0119] As described above, the information processing device 22 can comprehensively evaluate the model set for the identification units 52-1 to 52-N by using the comprehensive identification result obtained by integrating the identification results obtained in the identification units 52-1 to 52-N.

[0120] <Computer Configuration Example>

[0121] Next, the above series of processes (information processing methods) can be performed by hardware or software. In the case where the series of processes are performed by software, the program constituting the software is installed in a general-purpose computer or the like.

[0122] Figure 11 This is a block diagram illustrating an embodiment of a computer in which a program for performing the above-described series of processes is installed.

[0123] The program can be pre-recorded in a hard disk 105 or ROM 103, which is a recording medium built into the computer.

[0124] Alternatively, the program can be stored (recorded) in a removable recording medium 111 driven by drive 109. Such a removable recording medium 111 can be provided as so-called packaged software. Examples of removable recording media 111 include floppy disks, CD-ROMs, magneto-optical (MO) disks, DVDs, magnetic disks, semiconductor memory, etc.

[0125] Note that the program can be installed into the computer from the removable recording medium 111 as described above, or it can be downloaded into the computer and installed into the internal hard drive 105 via a communication network or broadcast network. That is, for example, the program can be wirelessly transmitted to the computer from a download site via a satellite used for digital satellite broadcasting, or it can be transmitted to the computer via a network (such as a local area network (LAN) or the Internet) over a wire.

[0126] The computer has a central processing unit (CPU) 102 built therein, and the CPU 102 is connected to the input / output interface 110 via bus 101.

[0127] When a user inputs a command via the input / output interface 110 through the operation input unit 107, the CPU 102 executes the program stored in the read-only memory (ROM) 103 according to the command. Alternatively, the CPU 102 loads the program stored in the hard disk 105 into the random access memory (RAM) 104 and executes the program.

[0128] Therefore, the CPU 102 performs processing according to the above flowchart or the processing configured in the above block diagram. Then, the CPU 102 outputs the processing result from the output unit 106, sends the processing result from the communication unit 108, or records the processing result to the hard disk 105 via the input / output interface 110, for example, as needed.

[0129] Note that the input unit 107 includes a keyboard, mouse, microphone, etc. Furthermore, the output unit 106 includes a liquid crystal display (LCD), speaker, etc.

[0130] In this specification, the processes executed by the computer according to the program are not necessarily executed sequentially in the order described in the flowchart. That is, the processes executed by the computer according to the program also include parallel or individual processes (e.g., parallel processing or processing by objects).

[0131] Furthermore, a program can be processed by a single computer (processor) or by multiple computers in a distributed manner. Moreover, the program can be transmitted to a remote computer and executed.

[0132] Furthermore, in this specification, a system refers to a collection of multiple components (devices, modules (parts), etc.), and it is irrelevant whether all components are housed in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a device in which multiple modules are housed in one housing, are both systems.

[0133] Furthermore, for example, a configuration described as a single device (or processing unit) can be partitioned and configured as multiple devices (or processing units). Conversely, a configuration described above as multiple devices (or processing units) can be collectively configured as a single device (or processing unit). Moreover, configurations other than those described above can, of course, be added to the configuration of each device (or each processing unit). Furthermore, as long as the overall system configuration or operation is substantially the same, a portion of the configuration of one device (or processing unit) can be included in the configuration of another device (or another processing unit).

[0134] Furthermore, for example, this technology can have a cloud computing configuration, in which one function is shared and collaboratively processed by multiple devices via a network.

[0135] Furthermore, for example, the above program can be executed on any device. In that case, only the device is required to have the necessary functions (function blocks, etc.) to obtain the necessary information.

[0136] Furthermore, for example, each step described in the flowchart above can be executed by one device, or it can be shared and executed by multiple devices. Moreover, when a step includes multiple processes, those multiple processes can be executed by one device, or they can be shared and executed by multiple devices. In other words, multiple processes included in a step can also be executed as processes of multiple steps. Conversely, processes described as multiple steps can be executed together as a single step.

[0137] It should be noted that in a computer-executed program, the processing of the steps describing the program can be executed sequentially in the order described herein, or in parallel, or individually at necessary time intervals (such as when a call is made). That is, the processing of the steps can be executed in a different order than described above, as long as there is no contradiction. Moreover, the processing of the steps describing this program can be executed in parallel with the processing of another program, or in combination with the processing of another program.

[0138] It should be noted that the various techniques described herein can be implemented independently as a single entity, as long as there is no contradiction. Of course, multiple arbitrary techniques can be combined to implement each other. For example, some or all of the techniques described in any embodiment can be combined with some or all of the techniques described in other embodiments. Furthermore, some or all of the aforementioned techniques can be combined with other techniques not described above.

[0139] <Application Example>

[0140] The technology disclosed herein can be applied to a variety of products. For example, the technology disclosed herein can be implemented as a device mounted on any type of mobile body, such as automobiles, electric vehicles, hybrid electric vehicles, motorcycles, bicycles, personal vehicles, airplanes, drones, ships, robots, construction machinery, or agricultural machinery (tractors).

[0141] Figure 12 This is a block diagram illustrating a schematic configuration example of a vehicle control system 7000, which serves as an example of a mobile body control system to which this technology can be applied. The vehicle control system 7000 includes multiple electronic control units connected via a communication network 7010. Figure 12 In the example shown, the vehicle control system 7000 includes a driving system control unit 7100, a body system control unit 7200, a battery control unit 7300, a vehicle external information detection unit 7400, a vehicle internal information detection unit 7500, and an integrated control unit 7600. The communication network 7010 connecting these multiple control units can be, for example, an in-vehicle communication network conforming to any standard (such as Controller Area Network (CAN), Local Interconnect Network (LIN), Local Area Network (LAN), or FlexRay (registered trademark)).

[0142] Each control unit includes a microcomputer that performs arithmetic processing according to various programs, a storage unit that stores programs to be executed by the microcomputer, parameters for various operations, etc., and drive circuits that drive various devices to be controlled. Each control unit includes a network I / F for communicating with other control units via a communication network 7010, and a communication I / F for communicating with devices, sensors, etc., inside or outside the vehicle via wired or wireless communication. Figure 12 In this diagram, the microcomputer 7610, general communication I / F 7620, dedicated communication I / F 7630, positioning unit 7640, beacon receiving unit 7650, vehicle-mounted device I / F 7660, audio-visual output unit 7670, vehicle-mounted network I / F 7680, and storage unit 7690 are shown as the functional configuration of the integrated control unit 7600. Other control units similarly include microcomputers, communication I / Fs, storage units, etc.

[0143] The driving system control unit 7100 controls the operation of devices related to the vehicle's driving system according to various programs. For example, the driving system control unit 7100 functions as a drive force generating device (such as an internal combustion engine or drive motor) for generating the vehicle's driving force, a drive force transmission mechanism for transmitting the driving force to the wheels, a steering mechanism for adjusting the vehicle's steering angle, and a braking device such as one for generating the vehicle's braking force. The driving system control unit 7100 may also function as a control device such as an anti-lock braking system (ABS) or electronic stability control (ESC).

[0144] The driving system control unit 7100 is connected to the vehicle state detection unit 7110. The vehicle state detection unit 7110 includes at least one of the following: a gyroscope sensor for detecting the angular velocity of the vehicle's axial rotational motion; an acceleration sensor for detecting the vehicle's acceleration; or sensors for detecting the amount of operation of the accelerator pedal, the amount of operation of the brake pedal, the steering angle of the steering wheel, engine speed, wheel speed, etc. The driving system control unit 7100 performs arithmetic processing using signals input from the vehicle state detection unit 7110 and controls the internal combustion engine, drive motor, electric power steering system, braking system, etc.

[0145] The body system control unit 7200 controls the operation of various devices installed on the vehicle body according to various programs. For example, the body system control unit 7200 is used as a control device for keyless entry systems, smart key systems, power window devices, or various lights (such as headlights, taillights, brake lights, hazard lights, or fog lights). In this case, radio waves or signals from various switches sent from portable devices that replace buttons can be input to the body system control unit 7200. The body system control unit 7200 accepts these radio wave or signal inputs and controls the vehicle's door locking devices, power window devices, lights, etc.

[0146] The battery control unit 7300 controls the secondary battery 7310, which serves as a power source for the drive motor, according to various programs. For example, information such as battery temperature, battery output voltage, or remaining battery capacity is input to the battery control unit 7300 from a battery device including the secondary battery 7310. The battery control unit 7300 uses these signals to perform arithmetic processing and to control the temperature adjustment of the secondary battery 7310 or the cooling devices included in the battery device.

[0147] The vehicle exterior information detection unit 7400 detects information about the exterior of the vehicle on which the vehicle control system 7000 is mounted. For example, the vehicle exterior information detection unit 7400 is connected to at least one of the image acquisition unit 7410 or the vehicle exterior information detector 7420. The image acquisition unit 7410 includes at least one of a time-of-flight (ToF) camera, a stereo camera, a monocular camera, an infrared camera, or other cameras. The vehicle exterior information detector 7420 includes at least one of, for example, an environmental sensor for detecting the current climate or weather, or an environmental information detection sensor for detecting another vehicle, obstacle, pedestrian, etc., around the vehicle on which the vehicle control system 7000 is mounted.

[0148] The environmental sensor can be at least one of the following: a rain sensor for detecting rain, a fog sensor for detecting fog, a sunlight sensor for detecting sunlight intensity, or a snow sensor for detecting snowfall. The ambient information detection sensor can be at least one of the following: an ultrasonic sensor, a radar device, or a laser imaging detection and ranging (LIDAR) device. The image acquisition unit 7410 and the vehicle exterior information detector 7420 can be provided as independent sensors or devices, or they can be provided as a device integrating multiple sensors or devices.

[0149] Here, Figure 13 An example of the mounting positions of the image acquisition unit 7410 and the vehicle exterior information detector 7420 is illustrated. Image acquisition units 7910, 7912, 7914, 7916, and 7918 are provided at at least one location, such as the front nose, side mirrors, rear bumper, rear door, or the upper part of the windshield inside the vehicle 7900. The image acquisition unit 7910 provided at the front nose and the image acquisition unit 7918 provided on the upper part of the windshield inside the vehicle primarily acquire images of the front of the vehicle 7900. The image acquisition units 7912 and 7914 provided on the side mirrors primarily acquire images of the sides of the vehicle 7900. The image acquisition unit 7916 provided on the rear bumper or rear door primarily acquires images of the rear of the vehicle 7900. The image acquisition unit 7918 provided on the upper part of the windshield inside the vehicle is mainly used to detect vehicles, pedestrians, obstacles, traffic lights, traffic signs, lanes, etc., ahead.

[0150] Note that, Figure 13Examples of the imaging ranges of the corresponding image acquisition units 7910, 7912, 7914, and 7916 are illustrated. Image acquisition range a indicates the image acquisition range of image acquisition unit 7910 provided at the front nose, image acquisition ranges b and c indicate the image acquisition ranges of image acquisition units 7912 and 7914 provided at the side mirrors, respectively, and image acquisition range d indicates the image acquisition range of image acquisition unit 7916 provided at the rear bumper or rear door. For example, a top-down view of the vehicle 7900 can be obtained by overlaying the image data captured by image acquisition units 7910, 7912, 7914, and 7916.

[0151] The vehicle exterior information detection units 7920, 7922, 7924, 7926, 7928, and 7930 provided at the front, rear, sides, and corners of vehicle 7900, as well as at the upper part of the windshield inside the vehicle, can be, for example, ultrasonic sensors or radar devices. The vehicle exterior information detection units 7920, 7926, and 7930 provided at the front nose, rear bumper, rear door, and the upper part of the windshield inside the vehicle 7900 can be, for example, LIDAR devices. These vehicle exterior information detection units 7920 to 7930 are primarily used to detect vehicles, pedestrians, obstacles, etc., ahead.

[0152] Will return Figure 12 Continuing the description. The vehicle exterior information detection unit 7400 enables the image acquisition unit 7410 to capture images of the exterior of the vehicle and receives the captured image data. Furthermore, the vehicle exterior information detection unit 7400 receives detection information from a connected vehicle exterior information detector 7420. If the vehicle exterior information detector 7420 is an ultrasonic sensor, radar device, or LIDAR device, the vehicle exterior information detection unit 7400 transmits ultrasonic waves, electromagnetic waves, etc., and receives information about the received reflected waves. The vehicle exterior information detection unit 7400 can perform object detection processing or distance detection processing based on the received information, such as detecting people, vehicles, obstacles, traffic signs, characters on the road surface, etc. The vehicle exterior information detection unit 7400 can perform environmental recognition processing based on the received information, such as recognizing rain, fog, road conditions, etc. The vehicle exterior information detection unit 7400 can calculate the distance to objects outside the vehicle based on the received information.

[0153] Furthermore, the vehicle exterior information detection unit 7400 can perform image recognition processing or distance detection processing based on the received image data to identify people, vehicles, obstacles, traffic signs, characters on the road surface, etc. The vehicle exterior information detection unit 7400 can perform processing such as distortion correction or alignment on the received image data and combine image data captured by different image acquisition units 7410 to generate a top-down view or panoramic image. The vehicle exterior information detection unit 7400 can use image data captured by different image acquisition units 7410 to perform viewpoint switching processing.

[0154] The vehicle interior information detection unit 7500 detects information about the interior of the vehicle. The vehicle interior information detection unit 7500 is connected, for example, to a driver state detection unit 7510 that detects the driver's state. The driver state detection unit 7510 may include a camera capturing an image of the driver, a biosensor detecting the driver's biological information, a microphone collecting sounds from inside the vehicle, etc. The biosensor is provided on, for example, seat surfaces, steering wheels, etc., and detects the biological information of occupants sitting in seats or drivers holding steering wheels. The vehicle interior information detection unit 7500 may calculate the driver's level of fatigue or concentration based on the detection information input from the driver state detection unit 7510, or may determine whether the driver is dozing off. The vehicle interior information detection unit 7500 may perform processing such as noise cancellation on the collected sound signals.

[0155] The integrated control unit 7600 controls the overall operation of the vehicle control system 7000 according to various programs. The integrated control unit 7600 is connected to the input unit 7800. The input unit 7800 is implemented, for example, by a device that can be operated by an occupant for input, such as a touch panel, button, microphone, switch, or lever. Data obtained by performing voice recognition on the sound input through the microphone can be input to the integrated control unit 7600. The input unit 7800 can be, for example, a remote control device using infrared or other radio waves, or it can be an external connection device such as a telephone or personal digital assistant (PDA) corresponding to the operation of the vehicle control system 7000. The input unit 7800 can be, for example, a camera, in which case the occupant can input information through gestures. Alternatively, data obtained by detecting the movement of a wearable device worn by the occupant can be input. Furthermore, the input unit 7800 may include, for example, an input control circuit that generates an input signal based on the information input by the occupant using the aforementioned input unit 7800 and outputs the input signal to the integrated control unit 7600. By operating the input unit 7800, occupants can input various data into the vehicle control system 7000 or give instructions on processing operations.

[0156] The storage unit 7690 may include a read-only memory (ROM) for storing various programs to be executed by the microcomputer, and a random access memory (RAM) for storing various parameters, calculation results, sensor values, etc. Furthermore, the storage unit 7690 may be implemented using a magnetic storage device (such as a hard disk drive (HDD), semiconductor storage device, optical storage device, magneto-optical storage device, etc.).

[0157] The Universal Communication I / F 7620 is a universal communication I / F that mediates communication with various devices existing in the external environment 7750. Cellular communication protocols (such as Global System for Mobile Communications (GSM), WiMAX, LTE, or LTE-Advanced (LTE-A)) or other wireless communication protocols (such as Wireless LAN, also known as Wi-Fi, or Bluetooth) can be installed on the Universal Communication I / F 7620. For example, the Universal Communication I / F 7620 can connect to devices (e.g., application servers or control servers) existing on external networks (e.g., the Internet, cloud networks, or operator-owned networks) via a base station or access point. Furthermore, for example, the Universal Communication I / F 7620 can connect to terminals existing near the vehicle (e.g., terminals of drivers, pedestrians, or shops, or machine-type communication (MTC) terminals) using peer-to-peer (P2P) technology.

[0158] The Dedicated Communication I / F 7630 is a communication I / F that supports communication protocols developed for use in vehicles. For example, standard protocols such as Wireless Access in a Vehicle Environment (WAVE) (which is a combination of lower-level IEEE 802.11p and upper-level IEEE 1609), Dedicated Short Range Communication (DSRC), or cellular communication protocols can be installed on the Dedicated Communication I / F 7630. Typically, the Dedicated Communication I / F 7630 performs V2X communication, which includes at least one of the following concepts: vehicle-to-vehicle communication, vehicle-to-infrastructure communication, vehicle-to-home communication, or vehicle-to-pedestrian communication.

[0159] The positioning unit 7640 receives GNSS signals, such as those from Global Navigation Satellite System (GNSS) satellites (e.g., GPS signals from Global Positioning System (GPS) satellites), performs positioning, and generates location information including the vehicle's latitude, longitude, and altitude. Note that the positioning unit 7640 can specify its current location by exchanging signals with a wireless access point, or it can obtain location information from a terminal such as a mobile phone, PHS device, or smartphone with positioning capabilities.

[0160] For example, the beacon receiving unit 7650 receives radio waves or electromagnetic waves transmitted from a wireless station installed on the road, and obtains information such as current location, traffic congestion, road closure, or estimated time. Note that the functionality of the beacon receiving unit 7650 can be included in the aforementioned dedicated communication I / F 7630.

[0161] The vehicle-mounted device I / F 7660 is a communication interface that mediates the connection between various vehicle-mounted devices 7760 and a microcomputer 7610 present in the vehicle. The vehicle-mounted device I / F 7660 can establish a wireless connection using wireless communication protocols such as Wireless LAN, Bluetooth (registered trademark), Near Field Communication (NFC), or Wireless USB (WUSB). Alternatively, the vehicle-mounted device I / F 7660 can establish wired communication via connection terminals (and, if necessary, cables) (not shown), such as Universal Serial Bus (USB), High Definition Multimedia Interface (HDMI) (registered trademark), or Mobile High Definition Link (MHL). The vehicle-mounted device 7760 may include, for example, a mobile device or wearable device owned by an occupant, or at least one of the information devices carried in or attached to the vehicle. Furthermore, the vehicle-mounted device 7760 may include a navigation device that searches for routes to any destination. The vehicle-mounted device I / F 7660 exchanges control signals or data signals with these vehicle-mounted devices 7760.

[0162] The vehicle network I / F 7680 is an interface that mediates communication between the microcomputer 7610 and the communication network 7010. The vehicle network I / F 7680 sends and receives signals according to predetermined protocols supported by the communication network 7010.

[0163] The microcomputer 7610 of the integrated control unit 7600 controls the vehicle control system 7000 according to various programs based on information acquired via at least one of the following: general communication I / F 7620, dedicated communication I / F 7630, positioning unit 7640, beacon receiving unit 7650, vehicle device I / F 7660, or vehicle network I / F 7680. For example, the microcomputer 7610 can calculate control target values ​​for the drive force generation device, steering mechanism, or braking device based on acquired information about the vehicle's interior and exterior, and output control commands to the driving system control unit 7100. For example, the microcomputer 7610 can perform coordinated control aimed at realizing the functions of an advanced driver assistance system (ADAS), including vehicle collision avoidance or shock absorption, distance-based following, speed maintenance, collision warning, lane departure warning, etc. Furthermore, the microcomputer 7610 can perform cooperative control for purposes such as autonomous driving, in which the vehicle controls the drive force generation device, steering mechanism, braking device, etc., based on acquired information about the vehicle's surroundings, and drives autonomously without relying on the driver's operation.

[0164] The microcomputer 7610 can generate three-dimensional distance information between the vehicle and objects such as surrounding structures or people based on information acquired via at least one of a general communication I / F 7620, a dedicated communication I / F 7630, a positioning unit 7640, a beacon receiving unit 7650, an in-vehicle device I / F 7660, or an in-vehicle network I / F 7680, and create local map information including surrounding information of the vehicle's current position. Furthermore, the microcomputer 7610 can predict hazards such as vehicle collisions, pedestrian approach, or entry into closed roads based on the acquired information and generate warning signals. These warning signals may, for example, be signals used to generate warning sounds or activate warning lights.

[0165] The audio-visual output unit 7670 sends at least one output signal of audio or image to an output device capable of visually or audibly notifying vehicle occupants or the outside of the vehicle. Figure 12 In the example, audio speaker 7710, display unit 7720, and dashboard 7730 are shown as output devices. Display unit 7720 may include at least one of, for example, an in-vehicle display or a head-up display. Display unit 7720 may have augmented reality (AR) display functionality. Output devices may be devices other than these, such as wearable devices, such as headphones or glasses-type displays worn by occupants, projectors, or lamps. When the output device is a display device, the display device visually displays the results obtained from various processes performed by microcomputer 7610 or information received from another control unit in various forms (such as text, images, tables, and graphics). Furthermore, when the output device is a sound output device, the sound output device converts audio signals, including reproduced sound data, acoustic data, etc., into analog signals and outputs the analog signals audibly.

[0166] exist Figure 12 In the example shown, it is noted that at least two control units connected via communication network 7010 can be integrated into a single control unit. Alternatively, each control unit may include multiple control units. Furthermore, the vehicle control system 7000 may include another control unit (not shown). Additionally, as described above, some or all of the functions performed by any control unit can be provided in another control unit. That is, predetermined computational processing can be performed by any control unit, as long as information is sent and received via communication network 7010. Similarly, sensors or devices connected to any control unit can be connected to another control unit, and multiple control units can send and receive detection information to each other via communication network 7010.

[0167] Note that this is used to implement according to the reference. Figure 5The computer program for each function of the information processing device 21 described in this embodiment can be installed on any control unit or the like. Furthermore, it is also possible to provide a computer-readable recording medium for storing such a computer program. Examples of recording media include magnetic disks, optical disks, magneto-optical disks, flash memory, etc. Additionally, the aforementioned computer program can be distributed via, for example, a network without using a recording medium.

[0168] In the aforementioned vehicle control system 7000, according to reference Figure 5 The information processing device 21 described in this embodiment can be applied to Figure 12 The integrated control unit 7600 is shown as an application example. For example, the image acquisition unit 31, domain adaptive processing execution unit 33, decision unit 36, and recognition processing unit 37 of the information processing device 21 correspond to the microcomputer 7610 of the integrated control unit 7600. Furthermore, for example, the storage unit 32 of the information processing device 21 corresponds to the storage unit 7690 of the integrated control unit 7600, and the sending unit 34 and receiving unit 35 of the information processing device 21 correspond to the vehicle network I / F 7680 of the integrated control unit 7600. For example, the integrated control unit 7600 can update the model of the recognizer by performing domain adaptive processing.

[0169] In addition, refer to Figure 5 At least some of the components of the described information processing device 21 can be used for Figure 12 This is implemented in a module (e.g., an integrated circuit module including a die) of the integrated control unit 7600 shown. Alternatively, they can be implemented by referring to... Figure 5 The described information processing device 21 and Figure 12 This is achieved through multiple control units in the vehicle control system 7000 shown.

[0170] <Example of configuration combinations>

[0171] Note that this technology can have the following configurations. (1)

[0173] An information processing device, comprising:

[0174] A predetermined number of recognition units, each equipped with a model updated by performing image recognition and unsupervised learning on a predetermined number of terminals; the recognition units are configured to perform image recognition on images that have already undergone image recognition on the predetermined number of terminals; and

[0175] The evaluation value calculation unit is configured to evaluate the recognition results in a predetermined number of recognition units and calculate an evaluation value for each recognition unit. (2)

[0177] According to the information processing device described in (1),

[0178] Learning is performed at predetermined intervals with predetermined image acquisition intervals in a predetermined number of terminals, and

[0179] The evaluation value calculation unit calculates the evaluation value for each constant time period longer than the predetermined interval. (3)

[0181] The information processing device according to (1) or (2) further includes:

[0182] The integration unit calculates the integrated recognition result obtained by combining the recognition results from a predetermined number of recognition units.

[0183] The evaluation value calculation unit calculates the evaluation value of each recognition unit by using the recognition results from a predetermined number of recognition units and the comprehensive recognition results. (4)

[0185] The information processing device according to any one of (1) to (3),

[0186] A predetermined number of recognition units perform image recognition on all images that have already undergone image recognition in a predetermined number of terminals. (5)

[0188] According to the information processing equipment described in (4),

[0189] A predetermined number of recognition units perform image recognition on images obtained by weighting the images of a predetermined terminal with additional information sent from that terminal. (6)

[0191] The information processing device according to (5) further includes:

[0192] The receiving unit receives images, models, and additional information transmitted from a predetermined number of terminals.

[0193] The receiving unit sets a predetermined number of terminal models to a corresponding predetermined number of recognition units, and supplies a predetermined number of images and a predetermined number of additional information to all predetermined number of recognition units. (7)

[0195] The information processing apparatus according to any one of (1) to (6) further includes:

[0196] The sending unit sends the evaluation values ​​calculated by the evaluation value calculation unit for the models set in a predetermined number of recognition units to the corresponding predetermined number of terminals. (8)

[0198] An information processing method, comprising:

[0199] Through information processing equipment,

[0200] In a predetermined number of recognition units, image recognition is performed on images that have already undergone image recognition in a predetermined number of terminals. For each recognition unit, a model is set up and updated by performing image recognition in the predetermined number of terminals and performing unsupervised learning.

[0201] The recognition results are evaluated in a predetermined number of recognition units, and an evaluation value is calculated for each recognition unit. (9)

[0203] An information processing device, comprising:

[0204] The execution unit is configured to perform unsupervised learning for a model of an image recognizer that performs image recognition on the image; and

[0205] The decision unit determines whether the learning process was performed correctly for the model updated by the learning in the execution unit, based on an evaluation value obtained by performing image recognition on a predetermined number of models that are different from each other.

[0206] If the decision unit determines that the learning process was not performed correctly, the execution unit will revert the model to the state it was before the learning process began. (10)

[0208] According to the information processing equipment described in (9),

[0209] The execution unit performs learning at predetermined intervals with image acquisition intervals, and calculates evaluation values ​​at each constant time interval longer than the predetermined interval. (11)

[0211] The information processing device according to (9) or (10) further includes:

[0212] The sending unit is configured to send predetermined additional information along with the model and image at the time when the evaluation value is obtained. (12)

[0214] The information processing apparatus according to any one of (9) to (11) further includes

[0215] The recognition processing unit sets the latest model updated by the execution unit in the recognizer, performs recognition processing on the image, and outputs the recognition result to the processing unit that uses the recognition result to perform predetermined processing. (13)

[0217] An information processing method, comprising:

[0218] Through information processing equipment,

[0219] For a model of an image recognizer that performs image recognition, unsupervised learning is performed; and

[0220] The evaluation value, obtained by performing image recognition on a predetermined number of different models, is used to determine whether the learning process was performed correctly for the model updated through the learning process.

[0221] If it is determined that the learning process was not performed correctly, the model will be reverted to the model before the learning process was performed. (14)

[0223] A system comprising:

[0224] The first information processing device has:

[0225] A predetermined number of recognition units, each equipped with a model updated by performing image recognition and unsupervised learning on a predetermined number of terminals; the recognition units are configured to perform image recognition on images that have already undergone image recognition on the predetermined number of terminals; and

[0226] An evaluation value calculation unit is configured to evaluate the recognition results in a predetermined number of recognition units and calculate an evaluation value for each recognition unit; and

[0227] The second information processing device has:

[0228] The execution unit is configured to perform unsupervised learning for a model of an image recognizer that performs image recognition on the image; and

[0229] The decision unit determines whether the learning process was performed correctly for the model updated by the learning in the execution unit, based on an evaluation value obtained by performing image recognition on a predetermined number of models that are different from each other.

[0230] If the decision unit determines that the learning process was not performed correctly, the execution unit will revert the model to the state it was before the learning process began.

[0231] It should be noted that this embodiment is not limited to the above embodiment, and various modifications can be made without departing from the spirit of this technology. Furthermore, the effects described herein are merely illustrative and not restrictive, and other effects can be obtained.

[0232] List of reference numerals

[0233] 11 Learning System

[0234] 12 Network

[0235] 13 vehicles

[0236] 14 servers

[0237] Information processing equipment 21 and 22

[0238] 31 Image Acquisition Unit

[0239] 32 storage units

[0240] 33 Domain Adaptive Processing Execution Unit

[0241] 34 Transmitting Unit

[0242] 35 Receiving Unit

[0243] 36 decision units

[0244] 37 Identification Processing Unit

[0245] 41 Environmental Characteristic Quantity Calculation Unit

[0246] 42 Domain Determination Unit

[0247] 43 Update value calculation unit

[0248] 51 Receiving Unit

[0249] 52 Identification Units

[0250] 53 Comprehensive Unit

[0251] 54 Evaluation Calculation Unit

[0252] 55 Transmitting Unit

Claims

1. An information processing device, comprising: A predetermined number of recognition units, each of which is equipped with a model updated by performing image recognition and unsupervised learning on images captured by a corresponding terminal in a predetermined number of terminals, and each recognition unit is configured to perform image recognition on all images uploaded by the predetermined number of terminals using the model set in the recognition unit. as well as An evaluation value calculation unit is configured to calculate, for each recognition unit, an evaluation value for individually evaluating the model set up in that recognition unit by using a comprehensive recognition result obtained from the predetermined number of recognition units combined with the recognition results in that recognition unit. Specifically, when the evaluation value of a certain identification unit is equal to or less than a predetermined value, feedback is sent to the terminal containing the model set in that identification unit, so that the terminal returns the model to the model before the update.

2. The information processing device according to claim 1, Learning is performed at predetermined intervals with predetermined image acquisition intervals in a predetermined number of terminals, and The evaluation value calculation unit calculates the evaluation value for each constant time period longer than the predetermined interval.

3. The information processing device according to claim 1, further comprising: The integration unit calculates the integrated recognition result obtained by integrating the recognition results from a predetermined number of recognition units.

4. The information processing device according to claim 1, A predetermined number of recognition units perform image recognition on images obtained by weighting the images of a predetermined terminal with additional information sent from that terminal.

5. The information processing device according to claim 4, further comprising: The receiving unit receives images, models, and additional information transmitted from a predetermined number of terminals. The receiving unit sets a predetermined number of terminal models to a corresponding predetermined number of recognition units, and supplies a predetermined number of images and a predetermined number of additional information to all predetermined number of recognition units.

6. The information processing device according to claim 1, further comprising: The sending unit sends the evaluation values ​​calculated by the evaluation value calculation unit for the models set in a predetermined number of recognition units to the corresponding predetermined number of terminals.

7. An information processing method, comprising: Through information processing equipment, In each of the predetermined number of recognition units, a model is set up that is uploaded by a corresponding terminal in the predetermined number of terminals, and updated by performing image recognition and unsupervised learning on the image captured by the corresponding terminal in the corresponding terminal. The model set up in the recognition unit is used to perform image recognition on all images uploaded by the predetermined number of terminals. as well as For each recognition unit, an evaluation value is calculated by combining the recognition results obtained from the predetermined number of recognition units with the overall recognition result and the recognition results within that recognition unit, to individually evaluate the model set up in that recognition unit. Specifically, when the evaluation value of a certain identification unit is equal to or less than a predetermined value, feedback is sent to the terminal containing the model set in that identification unit, so that the terminal returns the model to the model before the update.

8. An information processing device, comprising: An execution unit is configured to perform unsupervised learning on a model of a recognizer that performs image recognition on images. The updated model, obtained by performing image recognition on images captured by the information processing device and performing unsupervised learning, is uploaded to a server. The server then sets up the model in a corresponding recognition unit out of a predetermined number of recognition units, and performs image recognition on all images uploaded by the predetermined number of information processing devices using the model set in the corresponding recognition unit. For each recognition unit, an evaluation value is calculated to individually evaluate the model set up in that recognition unit using a composite recognition result obtained from the predetermined number of recognition units and the recognition result in that recognition unit. as well as The decision unit, based on the evaluation value calculated by the server, determines whether the learning process, which was implemented in the execution unit, has been performed correctly. If the evaluation value determined by the decision unit is equal to or less than the predetermined value, the execution unit will return the model to the model before the learning process.

9. The information processing device according to claim 8, The execution unit performs learning at predetermined intervals with image acquisition intervals, and calculates evaluation values ​​at each constant time interval longer than the predetermined interval.

10. The information processing device according to claim 8, further comprising: The sending unit is configured to send predetermined additional information along with the model and image at the time when the evaluation value is obtained.

11. The information processing device according to claim 8, further comprising: The recognition processing unit sets the latest model updated by the execution unit in the recognizer, performs recognition processing on the image, and outputs the recognition result to the processing unit that uses the recognition result to perform predetermined processing.

12. An information processing method, comprising: Through information processing equipment, For the model of the recognizer that performs image recognition on images, unsupervised learning is performed, wherein the model updated by performing image recognition on images captured by the information processing device and performing unsupervised learning is uploaded to the server, so that the server sets the model in the corresponding recognition unit in a predetermined number of recognition units, and performs image recognition on all images uploaded by the predetermined number of information processing devices using the model set in the corresponding recognition unit, and for each recognition unit, an evaluation value for individually evaluating the model set in that recognition unit is calculated by using the comprehensive recognition result obtained by the predetermined number of recognition units and the recognition result in that recognition unit; as well as Based on the evaluation value calculated by the server, it is determined whether the learning process was performed correctly for the model updated through the learning. If the evaluation value is determined to be equal to or less than the predetermined value, the model will be returned to the model before the learning process.

Images