Mobile object control system, information processor, method for mobile object control system, and method for generating machine learning model

Abstract

To provide a mobile object control system that can improve prediction accuracy when predicting an area on an image that corresponds to an instruction of a user including a relative positional relationship.SOLUTION: A mobile control system according to the present disclosure includes a memory and one or more processors. Instructions stored in the memory, when executed by the one or more processors, cause the one or more processors: to acquire an image; to acquire an instruction of a user in natural language that includes a relative positional relationship; and to use one or more machine learning models to predict, on the basis of a fusion feature quantity obtained by fusing an image feature quantity indicating a feature quantity of a scene captured in the image, a depth of the scene captured in the image, and a linguistic feature quantity indicating a linguistic feature quantity related to the instruction of the user, an area in the image corresponding to a position in the scene indicated by the instruction of the user.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to a mobile object control system, an information processing device, a method for a mobile object control system, and a method for generating a machine learning model. [Background technology]

[0002] In recent years, techniques for predicting specific regions within an image have become known, such as referring image segmentation, which predicts the region of a subject contained in an image, and visual grounding, which predicts specific regions within an image that correspond to instructions given in natural language.

[0003] Non-Patent Document 1 discloses a technology for recognizing objects in an image that are highly relevant to a user's utterance in natural language. In Non-Patent Document 1, a first heat map indicating pixels in an RGB image that are highly relevant to the utterance is combined with a second heat map indicating pixels in a depth image that are highly relevant to the utterance, and a clustering process is performed to identify objects that are highly relevant to the utterance. Non-Patent Document 2 discloses a technology for improving accuracy in the task of reference image segmentation by fusing image features and language features using an attention mechanism. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Fethiye Irmak Dogan and 1 other person, "Using Depth for Improving Referring Expression Comprehension in Real-World Environments", arXiv:2107.04658v1 [cs.RO], [online], July 9, 2021, retrieved January 18, 2024, Internet<URL:https: / / www.semanticscholar.org / reader / a66816dd86fdc3c5ecde919246d2ac18b08720b5> [Non-patent document 2] Zhao Yang and four others, "LAVT: Language-Aware Vision Transformer for Referring Image Segmentation," [online], retrieved January 18, 2024, Internet <URL https: / / openaccess.thecvf.com / content / CVPR2022 / papers / Yang_LAVT_Language-Aware_Vision_Transformer_for_Referring_Image_Segmentation_CVPR_2022_paper.pdf> Summary of the Invention [Problem to be solved by the invention]

[0005] However, when a user's instruction includes a relative positional relationship with a target, such as "in front of the car on the right," sufficient accuracy may not be obtained even if linguistic features are combined with image features of an RGB image.

[0006] The present invention has been made in consideration of the above-mentioned problems, and its purpose is to realize a technology that can improve prediction accuracy when predicting an area on an image that corresponds to a user instruction, including a relative positional relationship. [Means for solving the problem]

[0007] According to the present invention, A mobile object control system, Memory and one or more processors; The instructions stored in the memory, when executed by the one or more processors, cause the one or more processors to: Acquiring an image; acquiring a user's instruction in natural language including a relative positional relationship; A mobile object control system is provided that uses one or more machine learning models to predict an area in an image that corresponds to a position in the scene indicated by a user instruction, based on a fusion feature that combines image features that indicate features of the scene captured in the image, the depth of the scene captured in the image, and linguistic features that indicate linguistic features related to the user instruction. [Effects of the Invention]

[0008] According to the present invention, it is possible to improve the prediction accuracy when predicting an area on an image that corresponds to a user instruction including a relative positional relationship. [Brief explanation of the drawings]

[0009] [Figure 1A] FIG. 1 shows an example of the configuration of a moving body according to an embodiment. [Figure 1B] FIG. 2 shows an example of the configuration of a moving body according to an embodiment. [Figure 2] FIG. 1 is a block diagram showing an example of the configuration of a control system of a moving body according to an embodiment; [Figure 3] FIG. 1 is a diagram showing an example of the functional configuration of a control unit 130 according to an embodiment. [Figure 4] FIG. 1 is a diagram illustrating a configuration of a target area prediction unit according to an embodiment. [Figure 5] FIG. 10 is a diagram illustrating a process of training a machine learning model of a target area prediction unit according to an embodiment. [Figure 6] 1 is a flowchart showing a series of operations for training a machine learning model used in region prediction processing according to an embodiment. [Figure 7]1 is a flowchart showing a series of operations in region prediction processing according to an embodiment; [Figure 8] 1 is a flowchart showing a series of operations in an inference stage of a region prediction process according to an embodiment; DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention as claimed, and not all combinations of features described in the embodiments are necessarily essential to the invention. Two or more of the features described in the embodiments may be arbitrarily combined. Furthermore, the same reference numerals are used for the same or similar components, and redundant explanations will be omitted.

[0011] In the following embodiment, a neural network model as a machine learning model (described later) is described as being executed in an inference stage and a learning stage in a mobile object, such as micromobility, which is an example of a mobile object control system. However, this embodiment is not limited to this example, and the learning stage processing may be executed in an information processing server located on a cloud or an edge, which is an example of an information processing device. Furthermore, the mobile object control system may be a mobile object, a control device such as an ECU included in the mobile object, or an information processing server on a cloud for controlling the mobile object. That is, the inference stage processing of the machine learning model according to this embodiment may be executed in the mobile object or an information processing server on a cloud. Furthermore, the mobile object is not limited to micromobility, and may include autonomously driven robots, four-wheeled or two-wheeled passenger cars, work vehicles, etc.

[0012] In the following embodiments, an ultra-compact electric vehicle with a passenger capacity of about one person will be described as an example of a mobile body that is micromobility. However, micromobility may also include vehicles that travel together with people while carrying luggage instead of a person on board. Furthermore, this embodiment is not limited to examples in which the mobile body is an electric vehicle, and can also be applied to mobile bodies other than electric vehicles.

[0013] The mobile body 100 according to this embodiment recognizes a travel area and generates a route using images captured by the mobile body itself, without using a high-precision map, and travels autonomously according to the generated route. At this time, the mobile body 100 according to this embodiment executes a machine learning model that appropriately predicts an area on an image corresponding to a position specified by speech, for example, in order to appropriately move to a location specified by speech by a user.

[0014] <Configuration of moving body> An example of the configuration of the moving body 100 will be described with reference to Figures 1A and 1B. Figure 1A shows a side view of the moving body 100 according to this embodiment, and Figure 1B shows the internal configuration of the moving body 100. In the figures, arrow X indicates the front-to-rear direction of the moving body 100, with F indicating the front and R indicating the rear. Arrows Y and Z indicate the width direction (left-right direction) and up-down direction of the moving body 100.

[0015] The mobile object 100 is an electric autonomous vehicle that includes a propulsion unit 112 and uses a battery 113 as a main power source. The battery 113 is, for example, a secondary battery such as a lithium-ion battery, and the mobile object 100 is self-propelled by the propulsion unit 112 using power supplied from the battery 113. The propulsion unit 112 includes a pair of left and right drive wheels 120 that are front wheels, and one driven wheel 121 that is a rear wheel. Note that the example of the propulsion unit 112 shown in FIG. 1A is just one example, and the propulsion unit 112 may have other forms, such as a four-wheeled vehicle. Furthermore, the rear wheels are not limited to being driven wheels, and may be driven by a drive mechanism. The mobile object 100 includes, for example, a single seat 111, but may also include multiple seats.

[0016] The traveling unit 112 includes a drive mechanism 122. The drive mechanism 122 is a mechanism that uses motors 122a and 122b as drive sources to rotate the corresponding drive wheels 120. The drive mechanism 122 can move the mobile body 100 forward or backward by rotating each of the drive wheels 120. The drive mechanism 122 can also change the traveling direction of the mobile body 100 by generating a rotation difference between the motors 122a and 122b. The driven wheels 121 can rotate around the Z direction as a rotation axis.

[0017] The moving body 100 includes detection units 114 to 116 that detect targets around the moving body 100. The detection units 114 to 116 are a group of external sensors that monitor the periphery of the moving body 100. In this embodiment, the detection units 114 to 116 are all imaging devices that capture images of the periphery of the moving body 100, and include, for example, an optical system such as a lens and an image sensor. However, when acquiring depth information, which will be described later, radar or lidar (Light Detection and Ranging) may be used in addition to the imaging devices.

[0018] The detection unit 114 is, for example, a single imaging device disposed at the front of the moving body 100, and is mainly used to acquire images of the area in front of the moving body 100. The detection unit 114 may be configured with two imaging devices spaced apart in the Y direction. The detection units 115 are disposed on the left and right sides of the moving body 100, respectively, and are mainly used to acquire images of the areas to the sides of the moving body 100. The detection unit 116 is disposed at the rear of the moving body 100, and is mainly used to acquire images of the area behind the moving body 100. The moving body 100 does not necessarily have to include the detection units 115 and 116.

[0019] <Configuration example of a control system for a moving object> 2 is a block diagram of a control system of the mobile object 100. The mobile object 100 includes a control unit (ECU) 130. The control unit 130 includes one or more processors including a CPU or a GPU, a memory device such as a semiconductor memory, an interface with an external device, and the like. The memory device stores programs executed by the processor and various data used by the processor for processing (e.g., weight parameters of a trained machine learning model, etc.). Multiple sets of processors, memory devices, and interfaces may be provided for different functions of the mobile object 100 and configured to be able to communicate with each other.

[0020] The control unit 130 acquires the outputs (e.g., images) of the detection units 114 to 116, input information from the operation unit 131, and audio information input from the audio input device 133, and executes various processes. The control unit 130, for example, controls the motors 122a and 122b (controls the driving of the driving unit 112), controls the display of a display panel included in the operation unit 131, notifies the occupants of the moving object 100 by audio, and outputs information. Furthermore, as will be described later, the control unit 130 receives a user instruction in natural language, such as "in front of the car on the right," and executes a process (area prediction process) of predicting an area in an image corresponding to the position specified by the instruction. The area prediction process may be executed using one or more machine learning models (e.g., a deep neural network).

[0021] The voice input device 133 includes, for example, a microphone, and collects voices such as speeches of passengers (users) of the moving object 100. The GNSS (Global Navigation Satellite system) sensor 134 receives GNSS signals and detects the current position of the moving object 100.

[0022] The storage device 135 includes a non-volatile recording medium that stores various data. The storage device 135 may also store programs executed by the processor, data used by the processor for processing, etc. The storage device 135 may also store various parameters of the machine learning model executed by the control unit 130 (for example, trained weight parameters and hyperparameters of a deep neural network, etc.).

[0023] The communication device 136 is a communication device that can communicate with an external device (for example, a communication terminal 140 owned by a user or an information processing server) via wireless communication such as Wi-Fi or fifth generation mobile communication.

[0024] <Example of control unit functional configuration> Next, an example of the functional configuration of the control unit 130 according to this embodiment will be described with reference to FIG. 3. The functions of the components of the control unit 130 shown in FIG. 3 are realized, for example, by one or more processors of the control unit 130 executing a program stored in a memory or the like. Note that the example shown in FIG. 3 illustrates a case in which the control unit 130 includes both a target area prediction unit 303 and a learning processing unit 304. That is, the example shown in FIG. 3 illustrates a case in which the control unit 130 is capable of executing both an inference stage process using a trained machine learning model and a learning stage process for training the machine learning model. However, if the control unit 130 only performs the inference stage process using a trained machine learning model, the control unit 130 does not need to include the learning processing unit 304. In this case, the learning stage process for training the machine learning model is executed by another device.

[0025] When performing processing in the inference stage, the instruction acquisition unit 301 acquires user instructions input via the operation unit 131 or the voice input device 133. User instructions input via the voice input device 133 may be converted into spoken sentences written in natural language by voice recognition, or may be acquired as voice information including natural language utterances. Furthermore, the user instructions may be text written in natural language input via the operation unit 131. In either case, the user instructions are acquired as linguistic information including a position specification in natural language. The position specification includes a relative positional relationship with a target, such as "in front of the car on the right." When performing processing in the learning stage, the instruction acquisition unit 301 acquires user instructions included in training data, which will be described later.

[0026] When performing processing at the inference stage, the image information acquisition unit 302 acquires the outputs (images) of the detection units 114 to 116. When performing processing at the learning stage, the image information acquisition unit 302 acquires images included in training data, which will be described later.

[0027] The target area prediction unit 303 performs area prediction processing using a machine learning model, using linguistic information specifying a location from the instruction acquisition unit 301 and an image from the image information acquisition unit 302. The machine learning model may be configured with one or more machine learning models. When performing processing at the inference stage, the target area prediction unit 303 is executed using parameters of the trained machine learning model (for example, weight parameters of an optimized neural network).

[0028] The control unit 130 can recognize the position and shape of an obstacle, the driving area, etc., using image information in addition to the processing by the target area prediction unit 303. The position and shape of an obstacle ahead of the moving body 100, the driving area, the road structure, etc. may be recognized by applying a pre-trained machine learning model for image recognition (different from the model used for area prediction processing) to the image obtained from the detection unit 114, for example.

[0029] The learning processing unit 304 trains the machine learning model used in the target area prediction unit 303 to generate a trained machine learning model. The learning processing unit 304 calculates the value of a loss function based on the difference between the prediction result by the target area prediction unit 303 and the correct data for the prediction result. At this time, the machine learning model of the target area prediction unit 303 outputs the prediction result using parameters of the machine learning model in the middle of the learning stage (e.g., weight parameters of a neural network). The learning processing unit 304 changes the parameters of the machine learning model so as to reduce the value of the loss function. The learning processing unit 304 controls the learning stage processing so as to repeat prediction by the target area prediction unit 303, calculation of the value of the loss function, and change of the parameters of the machine learning model using training data.

[0030] The training data consists of multiple sets of datasets, each set consisting of an image, a user's instructions in natural language, including a location specification within the image, and correct answer data indicating an area within the image. The user's instructions in the correct answer data include various instructions indicating relative positional relationships with various targets, such as "in front of the car on the right." The relative positional relationships included in the correct answer data include various expressions representing the up / down direction or the left / right direction in the image plane. The relative positional relationships included in the correct answer data also include various expressions representing the front or back side of the image plane. Furthermore, the targets included in the correct answer data as base points for relative positions also include various targets. Such targets include, for example, movable targets such as pedestrians, bicycles, vehicles, and robots, as well as various expressions representing trees, buildings, traffic lights, or installed targets such as vending machines and postboxes, roads, intersections, etc. The images included in the correct answer data include images of various targets corresponding to the user's instructions, captured in various states.

[0031] The driving control unit 305 determines a driving route to a position corresponding to the instruction based on the area in the image corresponding to the instruction predicted by the target area prediction unit 303 and the driving area recognized using the image information, and determines a control amount for the mobile object according to the determined driving route. For example, if a user instructs the mobile object 100 to "stop in front of the car on the right," the target area prediction unit 303 predicts a position corresponding to "in front of the car on the right," determines a driving route to that position, and moves the mobile object 100 along that route. Note that the user's instruction does not have to be an instruction to stop the mobile object 100. For example, the user's instruction may be an instruction to "move in front of the car on the right," in which case the mobile object 100 may move toward the position corresponding to "in front of the car on the right." In either case, the driving control unit 305 is executed only when the target area prediction unit 303 performs processing in the inference stage. Any method may be used to determine a driving route using an area in the image as a target area, and known methods may be used. The driving control unit 305 further controls the driving of the mobile object 100 according to the determined control amount (e.g., controls the motors 122a and 122b).

[0032] <Operation of machine learning model used in region prediction processing> With reference to FIG. 4, a machine learning model used in the region prediction process according to this embodiment will be described.

[0033] The image 410 is an image (X img ), which is a captured image or an image included in training data. The image feature extraction unit 401 inputs the image 410 into a machine learning model and extracts image features of the scene captured in the image. The image features may be, for example, image features for each RGB channel. The image feature extraction unit 401 can extract image features by, for example, convolution or pooling processing, but the image feature extraction unit 401 may also extract image features by other configurations such as a transformer. The image feature extraction unit 401 extracts, for example, a feature map (F img) is output. If the unit area is 1x1 pixel, the feature map will be a map of the resolution of the input image (i.e., each pixel will have a feature).

[0034] The depth feature prediction unit 402 inputs the image 410 into a machine learning model to predict the depth of the scene captured in the image from the imaging device. The machine learning model may be, for example, a known machine learning model that can predict depth from a single image. The depth feature prediction unit 402 converts the depth feature encoded and decoded from the image 410 by the machine learning model into, for example, a depth map (F depth ) is output. The depth image 413 shown in FIG. 4 shows how the depth image with the resolution of the image 410 is divided into a grid corresponding to the unit area, in order to make the depth map easier to understand. If the unit area is 1×1 pixel, the depth map will be a map of the resolution of the input image.

[0035] Image feature map F img and the depth map F depth Since each of these has a feature amount or depth for each predetermined unit area, the target area prediction unit 303 generates a feature amount map F of the image as a two-dimensional map of the same size. img and the depth map F depth The target region prediction unit 303 can concatenate the feature amount map F of the image. img and the depth map F depth By concatenating and, we obtain a concatenated map F img+depth Generate.

[0036] The user instruction 411 is linguistic information acquired by the instruction acquisition unit 301, and is linguistic information from the voice input device 133 or the operation unit 131, or linguistic information included in the training data. The user instruction 411 includes a position specification in natural language including a relative positional relationship, such as "in front of the car on the right," and indicates a position within the captured scene. The specified position within the scene corresponds to a specific area within the image.

[0037] The linguistic feature extraction unit 403 may include a machine learning model using a transformer such as BERT, or a recursive machine learning model such as LSTM or GRU. The linguistic feature extraction unit 403 extracts linguistic features (linguistic features F lang The linguistic features may be encoded into a vector representation, for example, as used in word embedding.

[0038] The feature amount fusion unit 404 generates a feature amount (connection map) F img+depth and the linguistic feature F extracted by the linguistic feature extraction unit 403. lang The feature fusion unit 404 can generate fused features by any configuration. The feature fusion unit 404 may include, for example, a Pixel-Word Attention Module (PWAM). The PWAM may be implemented, for example, by a connection map (F img+depth ) is input as a query to the attention mechanism, and the language features are input as keys and values ​​to the attention mechanism. The language features are then fused into each unit region of the connected features to generate fused features (F fusion) is generated. In this way, by fusing (associating) language features with the features for each unit area in the concatenated map, it becomes possible to identify image features and depths that are highly correlated with language features. In other words, highly accurate predictions are possible that take into account the relationship between image features, depths, and language features. When a user instruction includes a relative positional relationship such as "in front of the car on the right," the image area corresponding to the position indicated by "in front of the car" can be predicted taking into account both image features and depths, thereby improving the accuracy of predictions for user instructions.

[0039] The prediction map generation unit 405 inputs the fusion feature generated by the feature fusion unit 404 into a machine learning model, and predicts an area in the image corresponding to a position in the scene indicated by a user instruction. The machine learning model of the prediction map generation unit 405 may be, for example, a decoder configured with a transformer. This decoder receives the fusion feature as input and outputs a prediction map indicating, for each region, the probability that the position in the scene indicated by the user's instruction is the position.

[0040] Alternatively, the machine learning model of the prediction map generation unit 405 may further include an encoder configured with a transformer. This encoder is, for example, an encoder that inputs a fusion feature and further encodes the fusion feature. That is, it further extracts a feature effective for the task (predicting a specified position) from the fusion feature. The encoded feature is then decoded by the above-mentioned transformer decoder to output a prediction map indicating, for each region, the probability that the position in the scene indicated by the user's instruction is the position. In the example shown in FIG. 4, for the sake of explanation, the prediction map output by the prediction map generation unit 405 is superimposed on the depth image 413. The region showing the highest probability on the prediction map is, for example, region 414. Note that the machine learning model included in the prediction map generation unit 405 may be configured with a model other than a transformer.

[0041] Next, with reference to FIG. 5, a process for training the machine learning model of the target area prediction unit 303 will be described. The learning stage process shown in FIG. 5 is executed by the learning processing unit 304. The machine learning model of the target area prediction unit 303 outputs a prediction map using parameters currently set in the learning stage. Note that the machine learning model of the target area prediction unit 303 executes processing by inputting an image and a user instruction included in training data. The prediction map includes a probability for each unit area, but for the sake of explanation, the example shown in FIG. 5 shows only the area 414 with the highest probability. The learning processing unit 304 uses a loss function to calculate a loss based on the difference between the prediction result in the prediction map and the area in the image indicated by the correct answer data (process 503). The loss function can be any function that uses the difference, but may be, for example, binary cross-entropy loss for calculating the loss in two-class classification. The correct answer data can be, for example, binary image data in which the correct answer area 502 on the map is assigned a value of "1" and other areas are assigned a value of "0." The learning processing unit 304 updates the parameters of the machine learning model of the target area prediction unit 303 so that the value of the loss function decreases (process 504). The learning processing unit 304 repeatedly executes the process by the target area prediction unit 303, process 503, and process 504 so that the value of the loss function becomes sufficiently small (for example, minimized), thereby training the machine learning model.

[0042] Next, a series of operations for training a machine learning model used in the region prediction process will be described with reference to Fig. 6. This process is realized by the control unit 130 expanding a program stored in the storage device 135 into the memory device of the control unit 130 and executing it. If the control unit 130 does not include the learning processing unit 304, the following process may be realized, for example, by one or more processors in an information processing server separate from the mobile object 100 executing the program. In this case, the information processing server realizes the operations of the instruction acquisition unit 301, image information acquisition unit 302, target region prediction unit 303, and learning processing unit 304 by executing the program in one or more processors.

[0043] In S601, for example, the instruction acquisition unit 301 and the image information acquisition unit 302 acquire the linguistic information of the training data (i.e., information including user instructions in natural language) and the image of the training data, respectively. Also, the learning processing unit 304 acquires the correct answer data corresponding to the training data (e.g., a map showing a specific area in the image).

[0044] In S602, the target region prediction unit 303 performs region prediction processing to predict a region in the image corresponding to a position in the scene indicated by a user instruction. As described above, the region prediction processing is realized by the machine learning model (the image feature extraction unit 401, the depth feature prediction unit 402, the language feature extraction unit 403, the feature fusion unit 404, and the prediction map generation unit 405). Details of this step will be described later.

[0045] In S603, the learning processing unit 304 calculates the value of the loss function based on the difference between the predicted area in the image and the area in the image in the correct answer data, as described above. When calculating the value of the loss function, the learning processing unit 304 may calculate the difference between the predicted area in the image and the area in the image indicated by the correct answer data only for the lower half of the image area. The reason for using the lower half of the image area is that when a user instructs a stopping position for a moving object, the target position is almost always in the lower half of the image area. In this case, limiting the objects for which the loss function is calculated can speed up processing.

[0046] In S604, the learning processing unit 304 determines whether processing for a group of data among the training data has been completed. If the learning processing unit 304 determines that processing for the group of data has not been completed, the process returns to S601 and the calculation of the loss function value using other data is repeated. If the learning processing unit 304 determines that processing for the group of data has been completed, the process proceeds to S605.

[0047] In S605, the learning processing unit 304 determines whether an optimization termination condition is met. The optimization processing condition may be any condition, but may include a predetermined number of iterations being repeated, the value of the loss function being reduced to a predetermined value or less, etc. If the learning processing unit 304 determines that the optimization termination condition is met, it ends this series of operations; if not, it proceeds to S606.

[0048] In S606, the learning processing unit 304 changes the parameters of one or more machine learning models so that the value of the loss function decreases (for example, based on the calculation result of backpropagation). The learning processing unit 304 then returns the process to S601.

[0049] In this way, it is possible to generate a machine learning model that improves prediction accuracy when predicting specific regions within an image that correspond to user instructions.

[0050] Next, a series of operations in the region prediction process will be described with reference to Fig. 7. This process is realized by the control unit 130 expanding a program stored in the storage device 135 into the memory device of the control unit 130 and executing it. Alternatively, the information processing server may realize the operation of the target region prediction unit 303 by executing the program with one or more processors. The processing by the machine learning model shown in S701 to S706 uses the parameters immediately before changed in S606.

[0051] In S701, the image feature extraction unit 401 extracts image feature values ​​indicating feature values ​​of the captured scene (i.e., feature values ​​of targets, roads, etc. in the captured image in RGB) from the image acquired in S601. The image feature extraction unit 401 extracts, for example, a feature value map F in which image feature values ​​are associated with each predetermined unit area (each area of ​​H×W pixels) of the image. img Output.

[0052] In S702, the depth feature prediction unit 402 predicts the depth of the captured scene from the image acquired in S601. As described above, the depth feature prediction unit 402 converts the depth features encoded and decoded from the image by the machine learning model into a depth map F depth The depth map is output as follows. For example, the depth is associated with each predetermined unit area (each area of ​​H×W pixels). The predetermined unit area may be 1×1 pixel, but by storing depth information for each predetermined unit area, the operation of the area prediction process can be speeded up. In S703, the language feature extraction unit 403 extracts language features included in the user's instruction acquired in S601.

[0053] In S704, the target region prediction unit 303 calculates the feature amount map F img and the depth map F depth By connecting the two, the linked map F img+depth The target region prediction unit 303 generates, for example, a feature amount map F img and the depth map Fdepth The feature values ​​and depths of the unit areas at corresponding positions are connected for each channel to generate a two-dimensional connected map.

[0054] In S705, the feature amount fusion unit 404 generates a feature amount (a link map F) in which the feature amount of the image and the depth are linked. img+depth ) and the linguistic feature F lang As described above, the feature fusion unit 404 generates a link map (F img+depth ) as a query for the attention mechanism, and the linguistic feature F lang are input as the key and value of the attention mechanism, and the fusion feature F is obtained by fusing the language feature into each unit region of the connected feature. fusion Generate.

[0055] In S706, the prediction map generation unit 405 adds the fusion feature F fusion The target area prediction unit 303 inputs the input data and predicts the area in the image corresponding to the position in the scene indicated by the user's instruction. After outputting the predicted area in the image, the target area prediction unit 303 returns the process to the caller.

[0056] Next, a series of operations in the inference stage of the machine learning model will be described with reference to Fig. 8. This process is realized by the control unit 130 expanding a program stored in the storage device 135 into the memory device of the control unit 130 and executing it. In this series of operations, the processes by the machine learning model shown in S701 to S706 use optimized parameters after the learning of the machine learning model is completed.

[0057] In S801, for example, the instruction acquisition unit 301 acquires, as language information (i.e., information including a location specification expressed in natural language), a user instruction input via the operation unit 131 or the voice input device 133. In addition, the image information acquisition unit 302 acquires an image from the detection unit 114.

[0058] In S701 to S706, the target region prediction unit 303 extracts image features, depth, and language features using the machine learning model (image feature extraction unit 401, depth feature prediction unit 402, language feature extraction unit 403, feature fusion unit 404, and prediction map generation unit 405) as described above, and fuses them to predict a specific region within the image that corresponds to a user instruction. When the target region prediction unit 303 outputs a prediction map including the predicted region within the image, it ends the series of operations in the region prediction process.

[0059] As described above, in the above-described embodiment, one or more machine learning models are used to predict an area in an image corresponding to a position in a scene indicated by a user's instruction based on fusion features obtained by fusing image features indicating the features of the scene captured in the image, the depth of the scene captured in the image, and language features indicating linguistic features related to the user's instruction. In this embodiment, by concatenating image features and depth for each predetermined unit area of ​​the image, the concatenated features and language features can be easily fused. By fusing language features with the concatenated features for each predetermined unit area, it becomes possible to identify an area in the image that has image features and depth that are highly correlated with the language features. In other words, highly accurate predictions are possible that take into account the relationship between image features, depth, and language features. In other words, it becomes possible to improve prediction accuracy when predicting an area in an image that corresponds to a user's instruction, including relative positional relationships.

[0060] <Summary of the embodiment> The above-described embodiments include an information processing device, an information processing method, a generation method, a learning model, a program, and a storage medium shown in the following items.

[0061] (Item 1) A mobile object control system, memory (e.g., 130); one or more processors (e.g., 130); The instructions stored in the memory, when executed by the one or more processors, cause the one or more processors to: acquiring an image (e.g., 302); Obtaining a user instruction in natural language that includes a relative positional relationship (e.g., 301); and predicting (e.g., 303, S602) an area in an image corresponding to a position in the scene indicated by a user instruction, based on fusion features obtained by fusing image features indicating features of the scene captured in the image, depth of the scene captured in the image, and linguistic features indicating linguistic features related to the user instruction, using one or more machine learning models.

[0062] According to this embodiment, when predicting an area on an image that corresponds to a user instruction including a relative positional relationship, it is possible to improve the prediction accuracy.

[0063] (Item 2) The instructions stored in the memory may cause the one or more processors to: 3. The mobile object control system according to claim 1, further comprising: using the one or more machine learning models to predict (e.g., 303) an area in the image corresponding to a position in the scene indicated by the user's instruction, based on fusion features obtained by fusing the image features, the depth, and the language features for each predetermined unit area of ​​the image.

[0064] According to this embodiment, by linking image features and depth for each predetermined unit area of ​​an image, it becomes possible to easily combine the linked features and language features.

[0065] (Item 3) The instructions stored in the memory may cause the one or more processors to: using the one or more machine learning models, extracting (e.g., 401) image features from the image that are indicative of features of a scene captured in the image; predicting (e.g., 402) from the image a depth of a scene captured in the image; Extracting (e.g., 403) linguistic features indicating linguistic features related to the user's instruction; and predicting (e.g., 405) an area in the image corresponding to a position in the scene indicated by the user's instruction based on a fusion feature obtained by fusing the image feature, the depth, and the language feature.

[0066] According to this embodiment, highly accurate prediction is possible by performing image feature extraction, depth prediction, language feature extraction, and region prediction using a machine learning model.

[0067] (Item 4) The instructions cause the one or more processors to: 4. The mobile object control system according to item 2 or 3, characterized in that the one or more machine learning models are used to concatenate the image features and the depth for each predetermined unit region of the image, and the concatenated features for each predetermined unit region are fused with the language features to generate the fused features (e.g., 404).

[0068] According to this embodiment, by combining language features with the concatenated features for each specified unit area, it becomes possible to identify positions within an image that have image features and depth that are highly correlated with the language features.

[0069] (Item 5) 5. The mobile control system of item 4, wherein the one or more machine learning models further include a Pixel-Wise Attention Mechanism (PWAM) that fuses the connected features for each of the predetermined unit regions with the linguistic features.

[0070] According to this embodiment, it is possible to accurately identify image features and depth, which are highly correlated with language features.

[0071] (Item 6) A mobile object control system, and one or more processors that execute the processing of the one or more machine learning models; the one or more machine learning models, a first machine learning model (e.g., 401) that extracts image features from the acquired image that indicate features of the scene captured in the image; a second machine learning model (e.g., 402) that predicts, from the image, the depth of the scene captured in the image; a third machine learning model (e.g., 403) that extracts linguistic features indicating linguistic features from a user's instruction in natural language that includes a relative positional relationship; and a fourth machine learning model (e.g., 405) that predicts an area in the image corresponding to a position in the scene indicated by the user's instruction based on a fusion feature that combines the image feature, the depth, and the language feature.

[0072] According to this embodiment, when predicting an area on an image that corresponds to a user instruction including a relative positional relationship, it is possible to improve the prediction accuracy.

[0073] (Item 7) An information processing device that trains one or more machine learning models, Memory and one or more processors; The instructions stored in the memory, when executed by the one or more processors, cause the one or more processors to: Acquiring an image, a user's instruction in natural language including a relative positional relationship, and ground truth data indicating an area within the image indicated by the user's instruction (e.g., 301, 302); using the one or more machine learning models to predict (e.g., 303) regions within the image that correspond to locations within the scene captured in the image indicated by the user's instructions, based on the image and the user's instructions; training the one or more machine learning models using a loss function based on a difference between the predicted region in the image and the region in the image represented by the ground truth data (e.g., 304); The information processing device is characterized in that the one or more machine learning models predict an area in the image corresponding to a position in the scene indicated by the user's instruction, based on a fusion feature that combines image features that indicate features of the scene captured in the image, a depth of the scene captured in the image, and linguistic features that indicate linguistic features related to the user's instruction.

[0074] According to this embodiment, when predicting an area on an image that corresponds to a user instruction including a relative positional relationship, it is possible to improve the prediction accuracy.

[0075] (Item 8) 8. The information processing device according to item 7, wherein the loss function includes a function for calculating binary cross-entropy loss.

[0076] According to this embodiment, it is possible to speed up the processing by performing two-class classification.

[0077] (Item 9) 9. The information processing device according to item 7 or 8, wherein training the one or more machine learning models includes using a loss function calculated in a lower half of the image area to calculate a difference between a predicted area in the image and an area in the image indicated by the user's instruction indicated by the correct answer data.

[0078] According to this embodiment, by limiting the targets for which the loss function is calculated, it is possible to speed up the processing.

[0079] (Item 10) An information processing device that trains one or more machine learning models, Memory and one or more processors; The instructions stored in the memory, when executed by the one or more processors, cause the one or more processors to: Acquiring an image, a user's instruction in natural language including a relative positional relationship, and ground truth data indicating an area within the image indicated by the user's instruction (e.g., 301, 302); using the one or more machine learning models to predict (e.g., 303) regions within the image that correspond to locations within the scene captured in the image indicated by the user's instructions, based on the image and the user's instructions; training the one or more machine learning models using a loss function based on a difference between a predicted region in the image and a region in the image indicated by the user's instruction as indicated by the ground truth data (e.g., 304); the one or more machine learning models, a first machine learning model (e.g., 401) that extracts image features from the acquired image that indicate features of the scene captured in the image; a second machine learning model (e.g., 402) that predicts, from the image, the depth of the scene captured in the image; A third machine learning model (e.g., 403) that extracts linguistic features indicating linguistic features from the user's instruction; and a fourth machine learning model (e.g., 404) that predicts an area in the image corresponding to a position in the scene indicated by the user's instruction based on a fusion feature that combines the image feature, the depth, and the language feature.

[0080] According to this embodiment, when predicting an area on an image that corresponds to a user instruction including a relative positional relationship, it is possible to improve the prediction accuracy.

[0081] (Item 11) 1. A method performed in a mobile control system, comprising: Acquiring an image; acquiring a user's instruction in natural language including a relative positional relationship; and predicting, using one or more machine learning models, an area in the image corresponding to a position in the scene indicated by the user's instruction, based on fusion features obtained by fusing image features indicating features of the scene captured in the image, a depth of the scene captured in the image, and linguistic features indicating linguistic features related to the user's instruction.

[0082] According to this embodiment, when predicting an area on an image that corresponds to a user instruction including a relative positional relationship, it is possible to improve the prediction accuracy.

[0083] (Item 12) 1. A method for generating one or more machine learning models, executed on an information processing device, comprising: acquiring an image, a user's instruction in natural language including a relative positional relationship, and correct answer data indicating an area within the image indicated by the user's instruction; using the one or more machine learning models to predict, based on the image and the user instructions, areas in the image that correspond to locations in the scene captured in the image indicated by the user instructions; and training the one or more machine learning models using a loss function based on a difference between a predicted region in the image and a region in the image indicated by the user's instruction indicated by the ground truth data; The method is characterized in that the one or more machine learning models predict an area in the image corresponding to a position in the scene indicated by the user's instruction, based on a fusion feature that combines image features that indicate features of the scene captured in the image, a depth of the scene captured in the image, and linguistic features that indicate linguistic features related to the user's instruction.

[0084] According to this embodiment, when predicting an area on an image that corresponds to a user instruction including a relative positional relationship, it is possible to improve the prediction accuracy.

[0085] The invention is not limited to the above-described embodiment, and various modifications and variations are possible within the scope of the gist of the invention.

Claims

1. A mobile object control system, Memory and one or more processors; The instructions stored in the memory, when executed by the one or more processors, cause the one or more processors to: Acquiring an image; acquiring a user's instruction in natural language including a relative positional relationship; and using one or more machine learning models, predicting an area in an image corresponding to a position in the scene indicated by a user's instruction, based on fusion features that combine image features indicating features of the scene captured in the image, the depth of the scene captured in the image, and linguistic features indicating linguistic features related to the user's instruction.

2. The instructions stored in the memory may cause the one or more processors to:

2. The mobile object control system of claim 1, wherein the one or more machine learning models are used to predict an area in the image corresponding to a position in the scene indicated by the user's instruction, based on fusion features obtained by fusing the image features, the depth, and the language features for each predetermined unit area of ​​the image.

3. The instructions stored in the memory may cause the one or more processors to: using the one or more machine learning models; extracting, from the image, image features indicative of features of a scene captured in the image; predicting from the image a depth of a scene captured in the image; extracting linguistic features indicating linguistic features related to the user's instruction; and predicting an area in the image corresponding to a position in the scene indicated by the user's instruction based on a fusion feature obtained by fusing the image feature, the depth, and the language feature.

4. The instructions cause the one or more processors to: The mobile object control system according to claim 2, characterized in that the one or more machine learning models are used to concatenate the image features and the depth for each predetermined unit region of the image, and the concatenated features for each predetermined unit region are fused with the language features to generate the fused features.

5. 5. The mobile object control system according to claim 4, wherein the one or more machine learning models further include a Pixel-Wise Attention Mechanism (PWAM) that fuses the linguistic features with the linked features for each of the predetermined unit regions.

6. A mobile object control system, one or more processors that execute the processing of the one or more machine learning models; the one or more machine learning models, a first machine learning model that extracts, from the acquired image, image features that indicate features of a scene captured in the image; a second machine learning model that predicts, from the image, a depth of a scene captured in the image; a third machine learning model that extracts linguistic features indicating linguistic features from a user's instruction in natural language that includes a relative positional relationship; and and a fourth machine learning model that predicts an area in the image corresponding to a position in the scene indicated by the user's instruction based on a fusion feature that combines the image feature, the depth, and the language feature.

7. An information processing device that trains one or more machine learning models, Memory and one or more processors; The instructions stored in the memory, when executed by the one or more processors, cause the one or more processors to: acquiring an image, a user's instruction in natural language including a relative positional relationship, and correct answer data indicating an area within the image indicated by the user's instruction; using the one or more machine learning models to predict, based on the image and the user instructions, areas within the image that correspond to locations within the scene captured in the image indicated by the user instructions; and training the one or more machine learning models using a loss function based on a difference between a predicted region in the image and a region in the image indicated by the ground truth data; The information processing device is characterized in that the one or more machine learning models predict an area in the image corresponding to a position in the scene indicated by the user's instruction based on a fusion feature that combines image features that indicate features of the scene captured in the image, the depth of the scene captured in the image, and linguistic features that indicate linguistic features related to the user's instruction.

8. The information processing device according to claim 7 , wherein the loss function includes a function for calculating a binary cross-entropy loss.

9. The information processing device according to claim 7, characterized in that training the one or more machine learning models includes using a loss function calculated as a difference between a predicted area in the image and an area in the image indicated by the user's instruction indicated by the correct answer data, in a lower half of the area of ​​the image.

10. An information processing device that trains one or more machine learning models, Memory and one or more processors; The instructions stored in the memory, when executed by the one or more processors, cause the one or more processors to: acquiring an image, a user's instruction in natural language including a relative positional relationship, and correct answer data indicating an area within the image indicated by the user's instruction; using the one or more machine learning models to predict, based on the image and the user instructions, areas within the image that correspond to locations within the scene captured in the image indicated by the user instructions; and training the one or more machine learning models using a loss function based on a difference between a predicted region in the image and a region in the image indicated by the user's instruction indicated by the ground truth data; the one or more machine learning models, a first machine learning model that extracts, from the acquired image, image features that indicate features of a scene captured in the image; a second machine learning model that predicts, from the image, a depth of a scene captured in the image; a third machine learning model that extracts linguistic features indicating linguistic features from the user's instruction; and a fourth machine learning model that predicts an area in the image corresponding to a position in the scene indicated by the user's instruction based on a fusion feature that combines the image feature, the depth, and the language feature.

11. 1. A method performed in a mobile control system, comprising: Acquiring an image; acquiring a user's instruction in natural language including a relative positional relationship; and predicting, using one or more machine learning models, an area in the image corresponding to a position in the scene indicated by the user's instruction, based on fusion features that combine image features indicating features of the scene captured in the image, a depth of the scene captured in the image, and linguistic features indicating linguistic features related to the user's instruction.

12. 1. A method for generating one or more machine learning models, executed on an information processing device, comprising: acquiring an image, a user's instruction in natural language including a relative positional relationship, and correct answer data indicating an area within the image indicated by the user's instruction; using the one or more machine learning models to predict, based on the image and the user instructions, areas within the image that correspond to locations within the scene captured in the image indicated by the user instructions; and training the one or more machine learning models using a loss function based on a difference between a predicted region in the image and a region in the image indicated by the user's instruction indicated by the ground truth data; The method is characterized in that the one or more machine learning models predict an area in the image corresponding to a position in the scene indicated by the user's instruction based on a fusion feature that combines image features that indicate characteristics of the scene captured in the image, the depth of the scene captured in the image, and linguistic features that indicate linguistic features related to the user's instruction.

Images