Information processing device for predicting region in image in response to speech, information processing method, learning method, and mobile body
The integration of image and language features using a trimodal encoder in the information processing device addresses the challenge of accurately identifying locations in images based on user utterances, improving navigation in autonomous mobility systems by predicting the correct area on an image corresponding to user instructions.
Patent Information
- Application Number
- PCT/JP2024/018220
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-17
- Filing Date
- 2024-05-16
- Publication Date
- 2024-11-21
AI Technical Summary
Current systems that classify objects using sensor data from LIDAR and cameras struggle to accurately identify a designated location in an image based on user utterances, as they do not effectively integrate language information with image data for precise location specification.
An information processing device equipped with trained machine learning models that combine image features, segmented image features, and language features using a trimodal encoder to predict the area on an image corresponding to a user's utterance, allowing for accurate location identification.
Enables the device to appropriately identify locations specified by user utterances within images, enhancing the precision of location-based navigation in autonomous mobility systems like micromobility vehicles.
Smart Images

Figure JP2024018220_21112024_PF_FP_ABST
Abstract
Description
Information processing device, information processing method, learning method, and mobile object for predicting an area in an image for speech
[0001] The present invention relates to an information processing device, an information processing method, a learning method, and a moving object that predict an area in an image for an utterance.
[0002] In recent years, there has been an increasing demand for ultra-small vehicles (micromobility) to support people's mobility within small areas. Micromobility vehicles include vehicles with a passenger capacity of around one person, and vehicles that travel alongside people while carrying luggage instead of a passenger, and require autonomous mobility technology.
[0003] Patent Document 1 discloses a system that classifies objects using sensor data including point cloud data from a LIDAR subsystem and image data from a camera subsystem, and performs fully or semi-automated driving using the classification results.
[0004] Japanese Patent Application Laid-Open No. 2020-32844
[0005] However, when using a mobile object, if a person verbally specifies a location where the mobile object should stop, there is a need for a technology that can appropriately identify the specified location using the speech and image data. The technology described in Patent Document 1 discloses a technology for classifying objects using point cloud data and image data, but does not take into consideration accepting a person's speech instruction and specifying a specified location in an image.
[0006] The present invention has been made in view of the above-mentioned problems, and an object of the present invention is to realize a technology that can appropriately identify a location related to an utterance contained in an image.
[0007] According to the present invention, there is provided an information processing device comprising: one or more processors that execute one or more trained machine learning models, wherein the one or more trained machine learning models include: an encoder that inputs text information including a location specification, a first image captured by an imaging device and including the location, and a second image in which the first image is divided into regions for each identical object, and outputs trimodal features generated to include image features of the captured first image, image features of the second image in which the regions are divided, and linguistic features of the text information; and a decoder that uses the trimodal features to output a region on the first image that corresponds to the location specification in the text information.
[0008] According to the present invention, it is possible to appropriately identify a location related to an utterance contained in an image.
[0009] Other features and advantages of the present invention will become apparent from the following description taken in conjunction with the accompanying drawings, in which the same or similar elements are designated by the same reference numerals.
[0010] The accompanying drawings are included in the specification and form a part thereof, illustrate embodiments of the present invention, and are used to explain the principles of the present invention together with the description thereof. A diagram showing an example of the configuration of a moving body according to an embodiment. (1) A diagram showing an example of the configuration of a moving body according to an embodiment. (2) A block diagram showing an example of the configuration of a control system of a moving body according to an embodiment. Block diagram showing an example of the functional configuration related to a control unit of a moving body according to an embodiment. A diagram showing an example of the configuration of a neural network model (machine learning model) used in mask prediction processing according to an embodiment. A diagram showing an example of the configuration of a trimodal encoder according to an embodiment. A diagram showing an example of the configuration of a hierarchical part of a model according to an embodiment. A flowchart showing a series of operations in mask prediction processing according to an embodiment. (1) A flowchart showing a series of operations in mask prediction processing according to an embodiment. (2) A flowchart showing a series of operations in model learning processing according to an embodiment.
[0011] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention as claimed, and not all combinations of features described in the embodiments are necessarily essential to the invention. Two or more of the features described in the embodiments may be arbitrarily combined. Furthermore, the same reference numerals are used for the same or similar components, and redundant explanations will be omitted.
[0012] In the following embodiment, a neural network model as a machine learning model, which will be described later, is described as an example case where the model is executed in a micromobility vehicle, which is an example of an information processing device. However, the machine learning model according to this embodiment is not limited to a mobile body such as a micromobility vehicle, and may be executed in an information processing server on the cloud, which is another example of an information processing device. Furthermore, the machine learning model according to this embodiment may be executed in another electronic device, which is yet another example of an information processing device, that is capable of acquiring captured images and speech.
[0013] In the following embodiments, an ultra-compact electric vehicle with a passenger capacity of about one person will be described as an example of a mobile body that is micromobility. However, micromobility may also include vehicles that travel together with people while carrying luggage instead of passengers. In addition, the present embodiments are not limited to examples in which the mobile body is an electric vehicle, and can also be applied to mobile bodies other than electric vehicles.
[0014] Mobile bodies such as the micromobility described above do not necessarily travel along specific, fixed routes. Furthermore, in order to be able to travel in both areas traveled by automobiles and areas traveled by pedestrians, they must travel in areas where high-precision maps are not available. Therefore, the mobile body 100 according to this embodiment recognizes the travel area and generates a route using images captured by the mobile body itself, without using high-precision maps, and autonomously travels according to the generated route. At this time, the mobile body 100 according to this embodiment executes a machine learning model that appropriately predicts the area on the image corresponding to the location specified by the user's utterance, for example, in order to appropriately travel to the location specified by the user's utterance.
[0015] <Configuration of Mobile Body> The configuration of mobile body 100 will be described with reference to Figures 1A and 1B. Figure 1A shows a side view of mobile body 100 according to this embodiment, and Figure 1B shows the internal configuration of mobile body 100. In the figures, arrow X indicates the front-to-rear direction of mobile body 100, F indicates the front, and R indicates the rear. Arrows Y and Z indicate the width direction (left-right direction) and up-down direction of mobile body 100.
[0016] The mobile body 100 is an electric autonomous vehicle equipped with a propulsion unit 112 and using a battery 113 as a main power source. The battery 113 is, for example, a secondary battery such as a lithium-ion battery, and the mobile body 100 is self-propelled by the propulsion unit 112 using power supplied from the battery 113. The propulsion unit 112 is equipped with a pair of left and right drive wheels 120 that are front wheels, and one driven wheel 121 that is a rear wheel. Note that the propulsion unit 112 may be in another form, such as a four-wheeled vehicle. Furthermore, the rear wheels are not limited to being driven wheels, and may be driven by a drive mechanism. The mobile body 100 is equipped with, for example, a single seat 111.
[0017] The traveling unit 112 includes a drive mechanism 122. The drive mechanism 122 uses motors 122a and 122b as drive sources to rotate the corresponding drive wheels 120. The drive mechanism 122 can move the mobile body 100 forward or backward by rotating each of the drive wheels 120. The drive mechanism 122 can also change the traveling direction of the mobile body 100 by generating a rotation difference between the motors 122a and 122b. The traveling unit 112 includes a driven wheel 121. The driven wheel can rotate around the Z direction as a rotation axis.
[0018] The moving body 100 is equipped with detection units 114 to 116 that detect targets around the moving body 100. The detection units 114 to 116 are a group of external sensors that monitor the periphery of the moving body 100. In the present embodiment, the detection units 114 to 116 are all imaging devices that capture images of the periphery of the moving body 100, and include, for example, an optical system such as a lens and an image sensor. However, in addition to the imaging devices, radar or lidar (light detection and ranging) may also be used.
[0019] The detection units 114 are arranged, for example, in pairs at the front of the moving body 100, spaced apart in the Y direction, and are mainly used to acquire images of the area in front of the moving body 100. The detection units 114 may be configured as a single imaging device. The detection units 115 are arranged on the left and right sides of the moving body 100, respectively, and are mainly used to acquire images of the areas to the sides of the moving body 100. The detection unit 116 is arranged at the rear of the moving body 100, and is mainly used to acquire images of the area behind the moving body 100. The moving body 100 does not necessarily have to include the detection units 115 and 116.
[0020] 2 is a block diagram of a control system of the mobile object 100. The mobile object 100 includes a control unit (ECU) 130. The control unit 130 includes one or more processors including a CPU or a GPU, a memory device such as a semiconductor memory, an interface with an external device, and the like. The memory device stores programs executed by the processor and various data used by the processor for processing (e.g., weight parameters of a trained model, etc.). Multiple sets of processors, memory devices, and interfaces may be provided for different functions of the mobile object 100 and configured to be able to communicate with each other.
[0021] The control unit 130 acquires the outputs (e.g., image information) of the detection units 114 to 116, input information from the operation unit 131, and audio information input from the audio input device 133, and executes various processes. The control unit 130, for example, controls the motors 122a and 122b (controls the driving of the driving unit 112), controls the display of the display panel included in the operation unit 131, and outputs audio alerts and information to the occupants of the vehicle 100. Furthermore, as will be described later, the control unit 130 executes a process (mask prediction process) of predicting an area in an image corresponding to a location specified by the user, using the output (e.g., image information) and audio information from the detection unit 114. The mask prediction process is executed using a machine learning model (e.g., a deep neural network).
[0022] The voice input device 133 includes, for example, a microphone, and collects voices of passengers (users) of the mobile object 100. The GNSS (Global Navigation Satellite system) sensor 134 receives GNSS signals and detects the current position of the mobile object 100.
[0023] The storage device 135 includes a non-volatile recording medium that stores various data. The storage device 135 may also store programs executed by the processor, data used by the processor for processing, etc. The storage device 135 may also store various parameters of the machine learning model executed by the control unit 130 (e.g., learned weight parameters and hyperparameters of a deep neural network, etc.).
[0024] The communication device 136 is a communication device capable of communicating with an external device (for example, a communication terminal 140 owned by a user) via wireless communication such as Wi-Fi or fifth generation mobile communication.
[0025] Next, an example of the functional configuration of the control unit 130 will be described with reference to FIG. 3. The user instruction acquisition unit 301 acquires user instructions input via the operation unit 131 or the voice input device 133. The user instructions may be converted into spoken text by voice recognition, or may be input as text. The user instructions may include, for example, a specification of a location where the mobile object 100 should arrive. The user instructions may also include an instruction to change the traveling direction of the mobile object 100, such as turning right or left, while the mobile object 100 is traveling. The image information acquisition unit 302 acquires the output (e.g., image information) of the detection units 114 to 116.
[0026] The target area prediction unit 303 performs mask prediction processing using the utterance specifying the location from the user instruction acquisition unit 301 and the image information from the image information acquisition unit 302. The target area prediction unit 303 performs mask prediction processing by executing a machine learning model, which will be described later.
[0027] In addition to the processing by the target area prediction unit 303, the control unit 130 may perform processing to recognize the position and shape of an obstacle, the driving area, etc., using image information. Recognition of the position and shape of an obstacle ahead of the moving body 100, the driving area, the road structure, etc. may be performed, for example, by applying a pre-trained machine learning model for image recognition (different from the model used for the mask prediction processing) to images obtained from the two detection units 114. The processing may also include estimating the depth from the moving body 100 using the images obtained from the two detection units 114 as stereo images.
[0028] The driving control unit 304 determines a driving route to the specified location based on the area in the image corresponding to the instruction predicted by the target area prediction unit 303 and the driving area recognized using the image information, and determines a control amount for the mobile object according to the determined driving route. Any method can be used to determine a driving route using the area in the image as a target position, and a known method may be used. The driving control unit 304 further controls the driving of the mobile object 100 according to the determined control amount (e.g., controls the motors 122a and 122b).
[0029] <Configuration of Model Used in Mask Prediction Processing> With reference to FIG. 4, a configuration example of a neural network model (machine learning model, also simply referred to as a learning model) used in mask prediction processing will be described.
[0030] The RoberTa module 401 receives an utterance text and calculates the feature of the utterance text (language feature h inst The spoken text is, for example, "Park in front of this white car," and the captured image X imgThe RoBERTa module 401 extracts language features using a BERT model trained by a known method proposed as RoBERTa, for example. Note that in this embodiment, an example is described in which the method proposed as RoBERTa ("RoBERTa: A Robustly Optimized BERT Pretraining Approach" (https: / / arxiv.org / abs / 1907.11692v1)) is used; however, language features may be extracted using a BERT model trained by another method or another model. Note that in this embodiment, an example is described in which spoken text is input; however, text input by a user may also be input to the RoBERTa module 401.
[0031] The Mask2Former module 403 generates a mask image X img and generate a semantic segmentation mask image X that divides the image into regions for each identical object. mask The Mask2Former module 403 generates a mask image (also simply referred to as a mask image) using a pre-trained model without fine-tuning for this mask prediction process (zero-shot generation). For generating the mask image, for example, a known model proposed as Mask2Former ("Masked-attention Mask Transformer for Universal Image Segmentation" (https: / / bowenc0221.github.io / mask2former / )) can be used, but is not limited to this. The mask image X according to this embodiment mask In the generation of the image, a case where the type of the divided area (for example, road, sky, vehicle, etc.) can be identified will be described as an example. However, this embodiment can also be applied to a case where the type of the area is not identified for some or all of the areas.
[0032] The trimodal encoder block 402 receives the captured image X img and the linguistic feature h inst and mask image Xmask and the trimodal feature ([F img ;F mask ]). The trimodal encoder block may be configured in a plurality of layers to handle multi-scale image features in which the spatial size is gradually reduced. The configuration of the trimodal encoder block in the layer lower than the trimodal encoder block 402 is included in the multi-layer block 420. The multi-layer block 420 will be described later with reference to FIG. 6.
[0033] The trimodal encoder block 402 includes a text image encoder block 404 and a text mask encoder block 405. The text image encoder block 404 uses an attention mechanism to encode the captured image X img The image features of the utterance text are added to the linguistic features of the utterance text. inst The multimodal features corresponding to the captured image (bimodal features F img ) is generated. In the following description, a feature obtained by combining two modal features, image feature and language feature, is called a bimodal feature to distinguish it from a trimodal feature. The text mask encoder block 405 also uses an attention mechanism to generate a mask image X mask The image features of the utterance text are added to the linguistic features of the utterance text. inst The multimodal features corresponding to the mask image (bimodal features F mask ) to generate the
[0034] Furthermore, the text image encoder block 404 generates a bimodal feature F img The image feature value V of the captured image img The image feature map E merged into img The text mask encoder block 405 outputs the bimodal feature F corresponding to the mask image. mask is the image feature of the mask image V mask The image feature map E merged into mask Output.
[0035] The trimodal encoder block 402 generates a bimodal feature F corresponding to the captured image. img and the bimodal feature F corresponding to the mask image. mask and the trimodal feature [F img ;F mask ]. The trimodal encoder block 402 outputs the bimodal feature F img and the bimodal feature F corresponding to the mask image. mask and may be output as a trimodal feature obtained by adding them for each matrix element.
[0036] As described above, in this embodiment, the trimodal encoder block 402 is provided with the text mask encoder block 405, and bimodal features are generated for the text mask encoder block 405. Furthermore, the trimodal encoder block 402 generates trimodal features including features related to the mask image. This makes it possible to use multimodal features including features of the segmentation mask in addition to image and language features.
[0037] 5 shows an example of the configuration of the trimodal encoder blocks 402, 601 to 603. FIG. 5 shows a more specific configuration of the text image encoder block 404 and the text mask encoder block 405. Note that when i=1, it corresponds to the trimodal encoder block 402, and when i=4, it corresponds to the trimodal encoder block 603. E img (i-1) corresponds to the captured image when i=1.
[0038] The PWAM (Pixel-Word Attention Module) 501 uses an attention mechanism to calculate the image feature V img The language feature h instThe PWAM 501 generates bimodal features by fusing the image features and the language features. For example, the PWAM 501 can use a known model (Pixel-Word Attention Module) described in "LAVT: Language-Aware Vision Transformer for Referring Image Segmentation" (https: / / openaccess.thecvf.com / content / CVPR2022 / papers / Yang_LAVT_Language-Aware_Vision_Transformer_for_Referring_Image_Segmentation_CVPR_2022_paper.pdf). The PWAM 501 inputs the image features as queries for the attention mechanism and inputs the language features as keys and values for the attention mechanism.
[0039] The Swin transformer block 502 receives an image (or image features) and extracts image features from the input information. The Swin transformer block 502 may be, for example, a transformer module described in "Swin Transformer: Hierarchical Vision Transformer using Shifted Windows" (https: / / openaccess.thecvf.com / content / ICCV2021 / papers / Liu_Swin_Transformer_Hierarchical_Vision_Transformer_Using_Shifted_Windows_ICCV_2021_paper.pdf). The Swin transformer block 502 corresponds to a transformer module corresponding to one scale for extracting multi-scale image features.
[0040] The LG (Language Gate) 505 adjusts the language features so that the adaptation language features are merged with the image features. The LG 505 learns a set of element-wise weight maps for the bimodal features so that the adaptation language features are merged with the image features. The bimodal features F processed by the LG 505 are imgThe image feature V output from the Swin transformer block 502 is img are merged (added) to obtain the image feature map E img is output as
[0041] The text image encoder block and the text mask encoder block may have the same internal module configuration, although they handle different image features. That is, the PWAM 503, the Swin transformer block 504, and the LG 506 may be the same modules as the PWAM 501, the Swin transformer block 502, and the LG 505, respectively.
[0042] In this manner, in this embodiment, trimodal features are generated using bimodal features corresponding to the captured image fused by PWAM 501 and bimodal features corresponding to the mask image fused by PWAM 503. This allows prediction of an area corresponding to an instruction to be made taking into account whether the area is a road area or a vehicle area. In other words, it becomes possible to select an area corresponding to an instruction from areas such as road areas where a vehicle can stop, thereby enabling more appropriate prediction of a segmentation mask.
[0043] Referring again to Figure 4, the Day-Night Classification branch 406 contains a classification model and uses trimodal features ([F img ;F mask ]) is used to classify the state of the subject represented in the captured image. Classification of the subject state using a classification model is performed as a subtask of the main task of predicting a specified region in the image.
[0044] Generally, the quality of a mask image may deteriorate during nighttime hours. Therefore, in this embodiment, when training a model for mask prediction processing, a loss function that partially includes the loss of the classification model is used. That is, the model for mask prediction processing is trained using the loss of a classification model that classifies between day and night, etc. This allows the model to be trained so that, under conditions where the quality of the mask image deteriorates (e.g., during nighttime hours), the bimodal feature corresponding to the mask image is not taken into account (or the degree of consideration is reduced). In other words, by using a classification model that classifies the state of the subject, the reliability of the input channel containing the mask image can be determined, and if the reliability of the input channel is low, the degree of consideration of that feature can be reduced. This makes it possible to prevent, for example, deterioration of prediction performance due to time of day.
[0045] Note that the day / night classification branch 406 uses the bimodal feature F corresponding to the mask image instead of the trimodal feature. mask In this way, the classification task can be performed without considering the bimodal features corresponding to the mask image under conditions where the quality of the mask image is degraded.
[0046] 4 illustrates an example in which the classification model classifies whether the subject is day or night. However, the classification model may be a model that classifies other subject states. The classification model may classify the subject state as at least one of whether the subject is day or night, the time of day, the weather, and whether or not there is sunshine.
[0047] In this embodiment, the output of the day / night classification branch 406 is used to calculate the loss function in the learning stage of the learning model (when the learning model is trained). However, the output of the day / night classification branch 406 may also be used in the inference stage of the learning model (when the trained model is executed). For example, the output of the day / night classification branch 406 may be input to the trimodal encoder blocks 601-603 and fused with features. Alternatively, the output of the day / night classification branch may be input to the trimodal decoder blocks 407, 611-613 and used in the decoding process.
[0048] The trimodal decoder block 407 decodes the features encoded by the trimodal encoder block to output an area on the image corresponding to a location specified by the utterance (predicted segmentation mask 408). The trimodal decoder block 407 incorporates (concatenates) the trimodal features into the decoding results of the lower layer, and thereby performs decoding in stages using the trimodal features including the image features of the mask image, thereby generating the predicted segmentation mask 408.
[0049] FIG. 6 shows a detailed configuration example of the multi-layer block 420. The multi-layer block 420 includes multiple tri-modal encoder blocks. As the processing progresses, it proceeds to lower layers, and in this embodiment, the fourth tri-modal encoder block is the block in the final layer. The lower the layer, the smaller the image feature amount handled by the encoder block in the lower layer. Therefore, the spatial size of the tri-modal feature amount output by the encoder block in the lower layer is smaller than the spatial size of the tri-modal feature amount output by the encoder block in the higher layer. In this way, it is possible to perform prediction processing using tri-modal features with different levels of spatial abstraction. The tri-modal feature amount [F img (i) ;F mask (i)] is input to a day / night classification branch provided for each layer, and the day / night classification branch outputs the classification result. However, the model according to this embodiment is not limited to this example. In other words, the spatial size of the trimodal features output by the encoder block of the lower layer may be the same as or larger than the spatial size of the trimodal features output by the encoder block of the higher layer.
[0050] The trimodal features [F img (i) ;F mask (i) ] is also input to the trimodal decoder block provided for each layer. The trimodal decoder block receives the feature H (i+1) The trimodal features encoded by the trimodal encoder block of the same layer are incorporated into the sigma-based feature set, and the incorporated features are decoded.
[0051] <Series of Operations in Mask Prediction Processing> Next, a series of operations in the mask prediction processing will be described with reference to Figures 7A and 7B. Note that this processing is realized as a function of the target area prediction unit 303 by the control unit 130 expanding a program stored in the storage device 135 into the memory device of the control unit 130 and executing it. Also, this processing is executed in a state where the learning model for the mask prediction processing described with reference to Figure 4 etc. has been trained. This processing is executed when the mobile object 100 acquires spoken text based on spoken voice including a location specification and acquires a captured image including the location.
[0052] In S701, the target area prediction unit 303 predicts the captured image (or E img (i-1) ) corresponding image feature V img (i) and the linguistic features of the spoken text to obtain bimodal features F img (i)When the layer of the trimodal encoder block is the highest layer, i=0, and the text image encoder block inputs the captured image. When i>0, the image feature map E img (i-1) i is incremented every time the process returns to S701.
[0053] In S702, the target region prediction unit 303 generates a mask image (or E mask (i-1) ) corresponding image feature V mask (i) and the linguistic features of the spoken text to obtain bimodal features F mask (i) Generate.
[0054] In S703, the target region prediction unit 303 calculates the bimodal feature F img (i) Based on this, the image feature map E img (i) For example, in the text image encoder block, LG505 generates the bimodal feature F img (i) Then, the bimodal features of the adjusted language features are used as image features V img (i) and merged to form an image feature map E img (i) is generated.
[0055] In S704, the target region prediction unit 303 calculates the bimodal feature F mask (i) Based on this, the image feature map E mask (i) For example, in the text mask encoder block, LG506 generates the bimodal feature F mask (i) Then, the bimodal features with the adjusted language features are used as image features V mask (i) and merged to form an image feature map Emask (i) is generated.
[0056] In S705, the target region prediction unit 303 calculates the fused bimodal feature F img (i) and F mask (i) and are combined to produce trimodal features [F img (i) ; F mask (i) In S706, the target region prediction unit 303 generates the trimodal feature [F img (i) ; F mask (i) ] is used to perform classification (e.g., day / night classification).
[0057] In S707, the target area prediction unit 303 determines whether i is the final layer. If the target area prediction unit 303 determines that i is the final layer (for example, the fourth layer shown in FIG. 6), the process proceeds to S708; otherwise, the process returns to S701.
[0058] In S708, the target region prediction unit 303 calculates the trimodal feature [F img (4) ; F mask (4) ] is decoded to H (4) The target region prediction unit 303 then passes the processing to the upper decoder.
[0059] In S709, the target region prediction unit 303 calculates the trimodal feature [F img (i-1) ; F mask (i-1) ] and the decoded result H (i) The value of i is decremented each time the process returns from step S711 to step S709.
[0060] In S710, the target region prediction unit 303 calculates the trimodal feature [F img (i-1) ; Fmask (i-1) ] and the decoded result H (i) The decoding process of the trimodal decoder can be expressed by, for example, equation (1).
[0061] Here, v represents upsampling, and ρ represents a process of, for example, applying a 3×3 convolution followed by batch normalization, then applying a 3×3 convolution again, followed by ReLU.
[0062] In S711, the target area prediction unit 303 determines whether i is the top layer. If the target area prediction unit 303 determines that i is the top layer, the process proceeds to S712; otherwise, the process returns to S709.
[0063] In S712, the target region prediction unit 303 outputs a region on the image corresponding to the specified location of the spoken text (the predicted segmentation mask 408). After outputting the predicted segmentation mask, the target region prediction unit 303 ends this process.
[0064] <Series of Operations Related to Training of the Learning Model> Next, a series of operations related to training of the learning model will be described with reference to FIG. 8 . Note that this process is realized, for example, by one or more processors in an information processing server separate from the mobile object 100 deploying a program stored in a non-volatile memory or the like into a memory device of the information processing server and executing it. Furthermore, this process is executed in a state where the learning model described with reference to FIG. 4 and the like has not yet been trained. This process trains the model using a large number of datasets, each set of which includes spoken text including a location specification, a test captured image, and ground truth data indicating the area of the correct predicted segmentation mask on the captured image. The datasets may be stored, for example, in a database provided in the information processing server.
[0065] In S801, the processor of the information processing server executes a mask prediction process (similar to S701 to S712 described in FIGS. 7A and 7B) using initial parameters. In S802, the processor of the information processing server acquires the position of the predicted segmentation mask and the classification result obtained in the mask prediction process.
[0066] In S803, the processor of the information processing server calculates a loss function based on the difference between the correct answer data and the position of the predicted segmentation mask and the class classification result. The loss function includes a first loss using the decoder output (the position of the predicted segmentation mask) and a correct answer indicating the area of the segmentation mask on the captured image, and a second loss using the classification result by the day / night classification model and a correct answer indicating the state of the subject (e.g., day / night). The function constituting the second loss is also referred to as an auxiliary loss function.
[0067] In S804, the processor of the information processing server updates the weight parameters of the neural network included in the learning model so as to decrease the value of the loss function. In S805, the processor of the information processing server determines whether a learning termination condition is met, and if it is determined that the termination condition is met, ends this process, otherwise returns to S802. The processor of the information processing server can determine whether a termination condition is met, such as whether the value of the loss function has converged to a value lower than a predetermined value and whether more trials have been performed than a predetermined number of epochs.
[0068] The information processing server transmits the parameters of the learning model obtained by learning (e.g., weight parameters of a neural network) to the mobile object 100, for example, via a network. In this way, the mobile object 100 can perform the above-mentioned mask prediction process using the learned learning model.
[0069] As described above, one or more trained machine learning models have an encoder that receives input of spoken text including a location specification, a first image captured by an imaging device and including the location, and a second image in which regions of the first image are divided for each identical object. In this case, the encoder uses an attention mechanism to generate bimodal features (F mask , F img ) and combine the generated bimodal features to generate trimodal features ([F img ;F mask ]). The one or more trained machine learning models include a decoder that uses trimodal features to output a region (408) on the first image that corresponds to the location specification in the spoken text. In this way, it is possible to appropriately identify a location related to the utterance contained in the image.
[0070] In the above-described embodiment, a model using three modal features, i.e., image, segmentation, and text, has been described as an example. However, the model according to this embodiment may use other information. For example, the model may handle features of environmental information (e.g., values acquired from a sensor equipped in a vehicle) as input. In other words, the model is not limited to trimodal features of image, segmentation, and text, but may also handle multimodal features (quadratic or higher) that include environmental information. Furthermore, from the perspective of learning, when inputting only part of the environmental information, the coefficient of the auxiliary loss term for the environmental information used in the input can be set to 0 (for example, if only sunlight can be input, the sunlight auxiliary loss term can be set to 0).
[0071] In addition, when the output of the day / night classification branch is used as the input for other blocks such as the trimodal decoder block, for example, when processing the inference stage using a model trained using the auxiliary loss function for day / night, the amount of sunlight obtained from a sensor equipped on the vehicle is calculated as p(y night ) (for example, if the amount of sunlight is high, p(y night ) is calculated using a transformation function that reduces the p(y night ) may be treated as an input to other blocks.
[0072] The above-described machine learning model may be executed by various types of information processing devices. For example, the information processing device may be the mobile object 100, or may be configured to be incorporated inside the mobile object 100 (i.e., the control unit 130). The information processing device may also be an information processing server that acquires images and sounds captured by the mobile object 100 and executes the above-described machine learning model. The computer program that operates the above-described machine learning model may also be a computer program that causes one or more computers to function as each means of the information processing device.
[0073] Summary of the embodiment 1. The information processing device of the above embodiment includes one or more processors that execute one or more trained machine learning models, and the one or more trained machine learning models are an encoder (e.g., 402) that receives as input text information including a location specification, a first image captured by an imaging device and including the location, and a second image obtained by dividing the first image into regions for each identical object, and the encoder generates trimodal features (e.g., [F img ;F mask ]); and a decoder that uses the trimodal features to output a region (e.g., 408) on the first image that corresponds to the location specification of the text information.
[0074] According to this embodiment, when predicting the area corresponding to the instruction, it is possible to make the prediction taking into consideration the area, such as the road area or the vehicle area, and thus it is possible to appropriately identify the location related to the utterance contained in the image.
[0075] 2. In the information processing device according to the above-described embodiment, the encoder includes a first encoder block (e.g., a text image encoder block) that outputs bimodal features corresponding to the first image by fusing image features of the first image with linguistic features of the text information using an attention mechanism, and a second encoder block (e.g., a text mask encoder block) that outputs bimodal features corresponding to the second image by fusing image features of the second image with linguistic features of the text information using an attention mechanism, and outputs the trimodal features by concatenating or adding the bimodal features corresponding to the first image and the bimodal features corresponding to the second image.
[0076] According to this embodiment, it is possible to appropriately combine language features with the image features of the captured image and the mask image using an attention mechanism, and it becomes possible to easily generate trimodal features.
[0077] 3. In the information processing device according to the above embodiment, the encoder generates a bimodal feature (e.g., F img ) into the image feature of the first image (for example, V img ) and merged image feature maps (e.g., E img ) and a bimodal feature corresponding to the second image (e.g., F mask ) to the image feature quantity (V mask ) and merged image feature maps (e.g., E mask ) and further output.
[0078] According to this embodiment, it is possible to generate features by appropriately merging language features with image features of each image.
[0079] 4. In the information processing device according to the above embodiment, the encoder is configured to: img ;F mask ]) and the image feature map (E img ) and the image feature map (e.g., E mask), and a second-layer encoder that generates bimodal features by fusing each image feature map input from the first-layer encoder with the linguistic features of the text information, and outputs trimodal features by combining the generated bimodal features.
[0080] According to this embodiment, it is possible to perform prediction processing using trimodal features that are generated in stages.
[0081] 5. In the information processing device according to the above-described embodiment, the spatial size of the trimodal feature output by the second-layer encoder is smaller than the spatial size of the trimodal feature output by the first-layer encoder.
[0082] According to this embodiment, it is possible to perform prediction processing using trimodal features with different levels of spatial abstraction.
[0083] 6. In the information processing device according to the above-described embodiment, the decoder includes: a second-layer decoder that performs decoding using the trimodal features output from the second-layer encoder; and a first-layer decoder that decodes features obtained by incorporating the trimodal features output from the first-layer encoder into the features decoded by the second-layer decoder.
[0084] According to this embodiment, the predicted segmentation mask 408 can be generated by performing step-by-step decoding using trimodal features including image features of the mask image.
[0085] 7. The information processing device according to the above-described embodiment further includes a classification model (for example, 406) that classifies the state of the subject represented by the first image as a subtask using the trimodal feature amount.
[0086] According to this embodiment, by using a classification model that classifies the state of the subject, the reliability of the input channel including the mask image can be determined, and if the reliability of the input channel is low, the degree of consideration of its features can be reduced.
[0087] 8. In the information processing device according to the above embodiment, a bimodal feature (e.g., F mask ) to classify the state of the object represented by the first image as a subtask.
[0088] According to this embodiment, by using a classification model that classifies the state of the subject, the reliability of the input channel including the mask image can be determined, and if the reliability of the input channel is low, the degree of consideration of its features can be reduced.
[0089] 9. The information processing device according to the above-described embodiment further includes a classification model that classifies states of the subject represented by the first image as subtasks using trimodal features generated by the second-layer encoder.
[0090] According to this embodiment, the reliability of an input channel including a mask image is determined based on trimodal features generated in stages, and if the reliability of the input channel is low, the degree of consideration of that feature can be reduced.
[0091] 10. In the information processing device according to the above embodiment, the bimodal feature (e.g., F mask ) to classify the state of the object represented by the first image as a subtask.
[0092] According to this embodiment, the reliability of an input channel including a mask image is determined based on bimodal features generated in stages, and if the reliability of the input channel is low, the degree of consideration of that feature can be reduced.
[0093] 11. In the information processing device according to the above-described embodiment, the classification model that classifies the state of the subject classifies the state of the subject into at least one of whether it is day or night, the time of day, the weather, and whether there is sunshine.
[0094] According to this embodiment, it is possible to determine the state of the subject under conditions where the reliability of the mask image may decrease depending on the time of day or the time of day.
[0095] 12. In the information processing device according to the above embodiment, the encoder generates a bimodal feature (e.g., F mask ) is processed by a language gate, and then the image feature of the second image (for example, V mask ) to obtain the image feature map (E mask ) is output (for example, S704).
[0096] According to this embodiment, the language features can be adjusted so that the language features of the adaptation amount are merged with the image features of the mask image.
[0097] 13. In the information processing device according to the above-described embodiment, the image feature of the first image is generated by inputting the first image to a transformer corresponding to the first image, and the image feature of the second image is generated by inputting the second image to a transformer corresponding to the second image.
[0098] According to this embodiment, by applying a transformer to each of the captured image and the mask image, highly accurate image feature amounts can be obtained for each.
[0099] 14. In the information processing device according to the above embodiment, the text information includes a voice utterance or input text.
[0100] According to this embodiment, the user's spoken voice or text entered by the user can be used.
[0101] 15. A learning method according to the above-described embodiment is a learning method executed by an information processing device, which generates one or more trained machine learning models by training one or more machine learning models, wherein each of the one or more machine learning models is configured as a neural network, and the one or more machine learning models are an encoder that receives as input text information including a location specification, a first image that is an image captured by an imaging device and includes the location, and a second image in which regions of the first image are partitioned for each identical object, and generates bimodal features (e.g., F) corresponding to each image obtained by fusing image features of each image with linguistic features of the text information using an attention mechanism. mask , F img ) and combine the generated bimodal features to generate trimodal features (e.g., [F img ;F mask ]), and a decoder that outputs a region on the first image that corresponds to the location specification of the text information, using the trimodal features or bimodal features corresponding to each image, wherein the learning method includes changing weight parameters of the neural network so that a value of a loss function that uses a loss using the output of the decoder and a correct answer indicating the region on the first image decreases.
[0102] According to this embodiment, it is possible to train a machine learning model that includes an encoder that extracts trimodal features and a decoder that outputs an area on an image that corresponds to a location specification from the extracted features.
[0103] 16. A learning method according to any of the above-described embodiments, wherein the one or more machine learning models further include a classification model that classifies, as a subtask, a state of the object represented by the first image, using features including image features of the second image or the trimodal features, and the learning method includes changing weight parameters of the neural network so that a value of a loss function that uses a loss using an output of the decoder and a correct answer indicating a region on the first image, and a loss using an output of the classification model and a correct answer indicating the state of the object, decreases.
[0104] According to this embodiment, under conditions where the quality of the mask image is degraded (for example, at night), the model can be trained so as not to consider (or to reduce the degree of consideration of) the bimodal feature corresponding to the mask image. In other words, by using a classification model that classifies the state of the subject, it is possible to generate a learning model that determines the reliability of an input channel that includes a mask image and reduces the degree of consideration of the feature when the reliability of the input channel is low.
[0105] The invention is not limited to the above-described embodiment, and various modifications and variations are possible within the scope of the gist of the invention.
[0106] This application claims priority to U.S. patent application Ser. No. 18 / 198,680, filed May 17, 2023, the entire contents of which are incorporated herein by reference.
Claims
1. An information processing device comprising: one or more processors that execute one or more trained machine learning models, the one or more trained machine learning models comprising: an encoder that inputs text information including a location specification, a first image captured by an imaging device and including the location, and a second image in which the first image is divided into regions for each identical object, and outputs trimodal features generated to include image features of the first captured image, image features of the second image in which the regions are divided, and linguistic features of the text information; and a decoder that uses the trimodal features to output a region on the first image that corresponds to the location specification in the text information.
2. The information processing device of claim 1, wherein the encoder includes a first encoder block that outputs bimodal features corresponding to the first image by fusing image features of the first image with language features of the text information using an attention mechanism, and a second encoder block that outputs bimodal features corresponding to the second image by fusing image features of the second image with language features of the text information using an attention mechanism, and the encoder outputs the trimodal features by concatenating or adding the bimodal features corresponding to the first image and the bimodal features corresponding to the second image.
3. The information processing device according to claim 2, wherein the encoder further outputs an image feature map in which bimodal features corresponding to the first image are merged with the image features of the first image, and an image feature map in which bimodal features corresponding to the second image are merged with the image features of the second image.
4. The information processing device described in claim 3, wherein the encoder includes: a first-layer encoder that outputs the trimodal features, the image feature map for the first image, and the image feature map for the second image; and a second-layer encoder that generates bimodal features by fusing each image feature map input from the first-layer encoder with language features of the text information, and outputs trimodal features by combining the generated bimodal features.
5. The information processing device according to claim 4, wherein a spatial size of the tri-modal feature output by the second-layer encoder is smaller than a spatial size of the tri-modal feature output by the first-layer encoder.
6. The information processing device according to claim 5, wherein the decoder comprises: a second layer decoder that performs decoding using the trimodal features output from the second layer encoder; and a first layer decoder that decodes features that incorporate the trimodal features output from the first layer encoder into features decoded by the second layer decoder.
7. The information processing device according to claim 1, further comprising a classification model that classifies a state of the subject represented by the first image as a subtask using the trimodal feature.
8. The information processing device according to claim 2, further comprising a classification model that classifies a state of a subject represented by the first image as a subtask using a bimodal feature corresponding to the second image.
9. The information processing device according to claim 4, further comprising a classification model that classifies the state of the subject represented by the first image as a subtask using trimodal features generated by the second layer encoder.
10. The information processing device according to claim 4, further comprising a classification model that classifies the state of the subject represented by the first image as a subtask using the bimodal features generated by the second layer encoder.
11. An information processing device as described in claim 7, wherein the classification model for classifying the state of the subject classifies the state of the subject into at least one of the following: day or night, the time of day, the weather, and whether or not there is sunshine.
12. The information processing device of claim 3, wherein the encoder performs language gate processing on bimodal features corresponding to the second image, and then adds the image features of the second image to output the image feature map for the second image.
13. An information processing device as described in claim 1, wherein the image features of the first image are generated by inputting the first image into a transformer corresponding to the first image, and the image features of the second image are generated by inputting the second image into a transformer corresponding to the second image.
14. The information processing device according to claim 1, wherein the text information includes spoken or input text.
15. A learning method executed on an information processing device, which generates one or more trained machine learning models by training one or more machine learning models, wherein each of the one or more machine learning models is composed of a neural network, the one or more machine learning models include: an encoder that inputs text information including a location specification, a first image captured by an imaging device and including the location, and a second image in which regions of the first image are divided for each identical object, the encoder generating bimodal features corresponding to each image by fusing image features of each image with linguistic features of the text information using an attention mechanism, and outputting trimodal features combining the generated bimodal features; and a decoder that outputs a region on the first image corresponding to the location specification in the text information using the trimodal features or the bimodal features corresponding to each image, wherein the learning method includes changing weight parameters of the neural network so that a value of a loss function using a loss using an output of the decoder and a correct answer indicating the region on the first image is reduced.
16. The learning method described in claim 15, wherein the one or more machine learning models further include a classification model that classifies a state of the subject represented by the first image as a subtask using features including image features of the second image or the trimodal features, and the learning method includes changing weight parameters of the neural network so that a value of a loss function using a loss using an output of the decoder and a correct answer indicating a region on the first image, and a loss using an output of the classification model and a correct answer indicating the state of the subject, is reduced.
17. A mobile body comprising: an imaging device; an acquisition means for acquiring text information; and one or more processors that execute processing of one or more trained machine learning models, wherein the one or more trained machine learning models are: an encoder that inputs the text information including a location specification, a first image captured by the imaging device and including the location, and a second image in which the first image is divided into areas for each identical object, and outputs trimodal features generated to include image features of the captured first image, image features of the second image in which the areas are divided, and language features of the text information; and a decoder that uses the trimodal features to output an area on the first image that corresponds to the location specification in the text information.
18. An information processing method executed on an information processing device, comprising: executing one or more trained machine learning models to input text information including a location specification, a first image captured by an imaging device and including the location, and a second image in which the first image is divided into regions for identical objects, and encoding trimodal features generated to include image features of the first captured image, image features of the second image in which the regions are divided, and language features of the text information; and decoding an area on the first image corresponding to the location specification in the text information using the trimodal features.
Citation Information
Patent Citations
Travel track generation method and travel track generation apparatus
JP2020032844A
Information processing apparatus, information processing method, learning method and moving object for predicting a region in an image corresponding to utterance
US12633101B2
Method, apparatus, electronic device, computer-readable storage medium, and computer program for image-based data processing
JP2020135852A
Object retrieval system, object retrieval method and learned model
JP2020190930A
Information processing method, information processing device, information processing system, and program
WO2022181130A1
Cited By
User intention-driven controllable image segmentation and semantic capture method
CN119904868A
