Response system and response generation method

Abstract

In this response system that uses a MultiModal Large Language Model (MM-LLM), the processing load can be reduced while improving the response quality. A response system (1) comprises: an acquisition unit (211) that acquires an input language and an input image from a user; a setting unit (221) that sets, in the input image, an important area corresponding to the input language; a first adjustment unit (222) that generates a first adjusted image by adjusting the input image on the basis of the important area; and a response generation unit (311) that uses a first language model to generate a response to the combination of the input language and the first adjusted image or to data corresponding to the combination thereof.

Description

Response system and response generation method

[0001] The present disclosure relates to a response system using a large-scale language model.

[0002] In recent years, progress has been made in the development of large language models (LLMs, hereinafter referred to as LLMs) that are primarily based on a structure called a Transformer. Furthermore, progress has also been made in the development of multi-modal large language models (MM-LLMs, hereinafter referred to as MM-LLMs), which are an extension of LLMs. MM-LLMs can generate responses to input language and input images. An example of MM-LLM technology is disclosed in Non-Patent Document 1.

[0003] MoE-LLaVA: Mixture of Experts for Large Vision-Language Models (https: / / arxiv.org / pdf / 2401.15947.pdf)

[0004] However, if the amount of input image data that the MM-LLM must interpret increases, the processing load on the MM-LLM increases, which increases the processing time required for the MM-LLM to respond after accepting the input language and input image.

[0005] In addition, there is generally a limit to the amount of data in an input image that an MM-LLM can process. Therefore, data thinning is performed on the input image so that the MM-LLM can process it. However, this thinning process reduces the amount of data in the area of ​​the input image that is related to the input language, which can lead to a decrease in the quality of the response by the MM-LLM.

[0006] One aspect of the present disclosure aims to realize a response system using MM-LLM that can reduce processing load while improving response quality.

[0007] A response system according to one aspect of the present disclosure includes an acquisition unit that acquires an input language and an input image from a user, a setting unit that sets an important area in the input image that corresponds to the input language, a first adjustment unit that generates a first adjusted image by adjusting the input image based on the important area, and a response generation unit that uses a first language model to generate a response to a combination of the input language and the first adjusted image or data corresponding to the combination.

[0008] A response generation method according to one aspect of the present disclosure includes an acquisition step of acquiring an input language and an input image from a user, a setting step of setting an important area in the input image that corresponds to the input language, a first adjustment step of generating a first adjusted image by adjusting the input image based on the important area, and a response generation step of generating a response to a combination of the input language and the first adjusted image or data corresponding to the combination using a first language model.

[0009] According to one aspect of the present disclosure, in a response system using MM-LLM, it is possible to reduce the processing load while improving the quality of the response.

[0010] 1 is a block diagram showing an example of the configuration of a response system of embodiment 1. FIG. 2 is a block diagram showing an example of a transmission data adjustment unit of embodiment 1. FIG. 3 is a sequence diagram showing an example of a processing flow in the response system of embodiment 1. FIG. 4 is a block diagram showing an example of the configuration of a response system of embodiment 2. FIG. 5 is a sequence diagram showing an example of a processing flow in the response system of embodiment 2. FIG. 6 is a block diagram showing an example of the configuration of a response system of embodiment 3. FIG. 7 is a block diagram showing an example of a transmission data adjustment unit of embodiment 3. FIG. 8 is a sequence diagram showing an example of a processing flow in the response system of embodiment 3. FIG. 9 is a block diagram showing an example of the configuration of a response system of embodiment 4. FIG. 10 is a block diagram showing examples of a transmission data adjustment unit and a local response generation unit of embodiment 4. FIG. 11 is a sequence diagram showing an example of a processing flow in the response system of embodiment 4. FIG. 11 is a block diagram showing an example of the configuration of a response system of embodiment 5. FIG. 12 is a sequence diagram showing an example of a processing flow in the response system of embodiment 5. FIG. 13 is a block diagram showing an example of the configuration of a response system of embodiment 6. FIG. 14 is a sequence diagram showing an example of a processing flow in the response system of embodiment 6.

[0011] [First Embodiment] <Overview of Response System> FIG. 1 is a block diagram showing an example of the configuration of a response system 1 according to the first embodiment. The response system 1 is a system configured using a first language model, and is a system that provides a response based on an input language and an input image input by a user. The input language and the input image are a query from a user requesting the generation of a response language for the input image. The first language model is, for example, MM-LLM. As shown in FIG. 1, the response system 1 includes, for example, a transmitting device 2, a receiving device 3, and a network 4. The transmitting device 2 and the receiving device 3 are connected to each other so as to be able to communicate with each other via the network 4.

[0012] The transmitting device 2 transmits multimodal instruction data (hereinafter referred to as MM instruction data) including at least an input language and a first adjusted image obtained by adjusting the input image to the receiving device 3, so that the receiving device 3 generates a response language in response to the input language and the input image. By receiving the MM instruction data, the receiving device 3 generates a response language for the combination of the input language and the first adjusted image using a first language model.

[0013] The response language generated by the receiving device 3 is presented by a presentation device. The presentation device may be any device that presents various types of information. For example, a display device and a speaker may be used as the presentation device. In this embodiment, the presentation device is described as the presentation unit 23 provided in the transmitting device 2, but the presentation device may be realized by a device other than the transmitting device 2 that is communicably connected to the transmitting device 2 and the receiving device 3. In this embodiment, the transmitting device 2 is a mobile terminal such as a smartphone or a tablet, and the receiving device 3 is a cloud server.

[0014] The transmission device 2 includes, for example, a first control unit 21 , an input unit 22 , a presentation unit 23 , a photographing unit 24 , and a first storage unit 25 .

[0015] The first control unit 21 performs overall control of the transmission device 2. The first control unit 21 includes, for example, an acquisition unit 211, a transmission data adjustment unit 212, and a presentation control unit 213. Specific functions of the first control unit 21 will be described later.

[0016] The input unit 22 accepts an input language and an input image from a user based on a user operation. The input unit 22 includes an input device such as an operation panel or a microphone that accepts user operation, and accepts the input language. The input unit 22 also accepts an input image captured by the capture unit 24 based on the user operation. Like the presentation unit 23, the input unit 22 may be realized by a device other than the transmission device 2 that is communicably connected to the transmission device 2.

[0017] The presentation unit 23 presents various types of information. The photographing unit 24 photographs an input image. The first storage unit 25 stores various programs used by the first control unit 21 and the various types of information described above.

[0018] The receiving device 3 includes, for example, a second control unit 31 and a second storage unit 32. The second control unit 31 performs overall control of the receiving device 3. The second control unit 31 includes, for example, a response generation unit 311. Specific functions of the second control unit 31 will be described later. The second storage unit 32 stores various programs used by the second control unit 31 and the various pieces of information described above. The second storage unit 32 stores, for example, a first language model 321.

[0019] As described above, in a conventional response system using an MM-LLM, an increase in the processing load of the MM-LLM can result in an increase in processing time (processing delay). Also, in this response system, a decrease in the amount of data in the area related to the input language in the input image can result in a decrease in the quality of the response language generated by the MM-LLM.

[0020] In the response system 1, the transmission data adjustment unit 212 sets the above-mentioned region as an important region and adjusts the input image taking the important region into consideration so that the data volume of the adjusted first adjusted image is reduced. Therefore, in the response system 1 using the first language model 321 (MM-LLM), it is possible to reduce the processing load while improving the quality of the response language.

[0021] Furthermore, when the first language model 321 is provided on a cloud server and the input language and input image are transmitted to the cloud server from a remote location, a network communication bandwidth (transmission path) that enables this transmission is required. As described above, in the response system 1, the amount of data of the adjusted first adjusted image can be reduced by the processing of the transmission data adjustment unit 212. This reduces the possibility of increasing the communication bandwidth, and also reduces the possibility of delays occurring due to image transmission even if the communication bandwidth is narrow.

[0022] An example of processing in the response system 1 that can achieve these effects will be described in detail below.

[0023] <Processing Flow> An example of the processing flow in the response system 1 will be described with reference to Figures 1 to 3. Figure 2 is a block diagram showing an example of the transmission data adjustment unit 212 according to the first embodiment. Figure 3 is a sequence diagram showing an example of the processing flow in the response system 1.

[0024] 3 , in the transmission device 2, when a user inputs an input language and an input image to the input unit 22, the acquisition unit 211 acquires the input language and input image accepted by the input unit 22 via the input unit 22 (S1; acquisition step). The acquisition unit 211 transmits the acquired input language and input image to the transmission data adjustment unit 212. As shown in FIG. 2 , the acquisition unit 211 transmits the acquired input language and input image to the setting unit 221, transmits the acquired input image to the first adjustment unit 222, and transmits the acquired input language to the transmission control unit 223.

[0025] Next, the setting unit 221 sets an important region in the input image corresponding to the input language based on the input language and the input image acquired in S1 (S2; setting step). As shown in Fig. 2, the setting unit 221 transmits data indicating the set important region to the first adjustment unit 222.

[0026] The setting unit 221 sets the important region using, for example, any of the methods in the following setting examples 1 to 6, or at least two or more of the setting examples 1 to 6. Through the processing of the setting unit 221, the first adjustment unit 222 can generate a first adjusted image that includes an important region including an image of the target object and that has a reduced amount of data from the input image.

[0027] (Setting Example 1) The setting unit 221 sets an important region using demonstrative pronouns (such as "that," "this," and "that") included in the input language.

[0028] For example, consider a case where the input language includes demonstrative pronouns (distant pronouns) such as "that," "that one," and "over there" that indicate an image of an object of interest that is relatively far from the image capturing unit 24 in the input image, such as "What's that?" In this case, the setting unit 221 sets, as the important region, an area that includes an image of the object of interest (distant object) that is estimated to be located far away in the input image. The object of interest is an object (e.g., an apple, an orange, etc.) included in the input image that is indicated by the demonstrative pronoun included in the input language.

[0029] Note that words and phrases included in the input language, such as demonstrative pronouns, can be extracted by using, for example, a known text analysis technique (language analysis process) that analyzes a sentence, and images of objects included in the input image, such as a target of interest, can be extracted by using, for example, a known image analysis technique or image recognition technique (object recognition technique).

[0030] The setting unit 221, for example, sets a relatively small region in the input image as the important region. For example, the setting unit 221 detects a rectangular region that includes an image of an object included in the input image, and sets a region whose area ratio to the entire input image is one-fourth or less as the important region including the distant target object. The ratio of the area of ​​the rectangular region to the entire input image can be adjusted arbitrarily through experiments, etc. Furthermore, for example, the setting unit 221 may set the rectangular region with the smallest area among all rectangular regions detected in the input image as the important region including the distant target object.

[0031] Also, consider a case where the input language includes demonstrative pronouns (proximate pronouns) such as "this," "this," and "here," which indicate an image of an object of interest that is relatively close to the image capturing unit 24 in the input image, such as "What is this?" In this case, the setting unit 221 sets, as the important region, an area that includes an image of the object of interest (nearby object) that is estimated to be located nearby in the input image.

[0032] The setting unit 221, for example, sets a relatively large region in the input image as the important region. For example, the setting unit 221 detects a rectangular region that includes an image of an object included in the input image, and sets a region in which the ratio of the area of ​​the rectangular region to the entire input image is equal to or greater than half as the important region including the near target object. The ratio of the area of ​​the rectangular region to the entire input image can be adjusted arbitrarily through experiments, etc. Furthermore, for example, the setting unit 221 may set the rectangular region with the largest area among all rectangular regions detected in the input image as the important region including the near target object.

[0033] Consider a case where the input language includes demonstrative pronouns (middle pronouns) such as "that," "that," and "over there" that indicate an image of an object of interest near a person or a robot in the input image, such as "What is that?" In this case, the setting unit 221 sets an area including an image of the object of interest (middle object) that is estimated to be a person or a robot as the important area. The robot is, for example, a conversational robot equipped with a conversation function.

[0034] The setting unit 221, for example, detects an area that includes at least a part of the image of a person or robot included in the input image, and sets that area, an area included in that area, or an area adjacent to that area as an important area that includes the central target object.

[0035] (Setting Example 2) The setting unit 221 sets an important region by using an image that is included in the input image and points to a target of interest.

[0036] In this case, the setting unit 221 detects an area in the input image that includes an image pointing to the target of interest. Examples of images pointing to the target of interest include images that provide directional clues that indicate the direction in which to point at something, such as a finger or a pointer. Examples of images pointing to the target of interest also include images that provide pointing clues that indicate something, such as the point of a laser pointer.

[0037] When the setting unit 221 detects an image indicating a direction indication clue in the input image, it sets an area including an image of the target of interest that is included in an area adjacent to the image of the direction indication clue in the direction indicated by the direction indication clue as an important area.When the setting unit 221 detects an image indicating a pointing clue in the input image, it sets an area including the image of the pointing clue and the image of the target of interest as an important area.

[0038] According to setting example 2, for example, when a user points to an object of interest and inputs the input language "What is this?", an area including the image of the object of interest that is located on the extension of the fingertip, or an area including the point and the image of the object of interest, can be set as an important area.

[0039] The extraction of images showing directional clues, the analysis of their directionality, and the extraction of images showing pointing clues can be performed using known image analysis techniques or image recognition techniques.

[0040] (Setting Example 3) The setting unit 221 sets an important region using language included in the input language that indicates the position of an object of interest. In setting example 3, the setting unit 221 sets an important region using image position designation information included in the input language. The image position designation information is language that designates, for example, a partial region of the input image from a planar perspective as an area in the input image that includes an image of the object of interest.

[0041] The setting unit 221 detects, from the input language, a language indicating a position on a plane in the input image, and sets an important region based on the detection result. Examples of the language include "right," "left," "top," "bottom," and "center." The language may also indicate a composite position of multiple positions, such as "top right" and "bottom left."

[0042] For example, when the setting unit 221 detects "right" from the input language, it sets an area that includes at least a portion of the area to the right of the center of the input image as the important area. For example, the setting unit 221 sets the right half of the input image as the important area. Similarly, when the setting unit 221 detects "left," "upper," and "lower" from the input language, it sets the areas corresponding to these languages ​​as the important area. When the setting unit 221 detects "center" from the input language, it sets an area of ​​a predetermined size that includes the center of the input image as the important area. For example, it sets an area that has a height half the height of the input image and a width half the width of the input image, and whose center coincides with the center of the input image, as the important area. The shape and size of the area corresponding to the detected language can be adjusted as desired.

[0043] Consider a case where the setting unit 221 detects, from the input language, a language indicating a position in the input image, as well as a reference language that serves as a reference for the position. In this case, the setting unit 221 may specify, in the input image, the position indicated by the above language as a relative position from an image of an object corresponding to the reference indicated by the above reference language, and set an important region. For example, if the setting unit 221 detects the language "to the left of the tree" from the input language, it specifies "tree" as the reference language (position reference). Then, the setting unit 221 detects an image of a "tree" in the input image and sets an area adjacent to the left side of the image of the "tree" as an important region. The setting unit 221 can determine whether the input language includes a reference language and a language indicating a position, for example, by using a known text analysis technique.

[0044] (Setting Example 4) The setting unit 221 sets an important region using language included in the input language that indicates the position of an object of interest. In setting example 4, the setting unit 221 sets an important region using 3D (dimensional) position designation information included in the input language. The 3D position designation information is language that designates, for example, a partial region of the input image from a spatial perspective as an area that includes a position in the input image that includes an image of the object of interest.

[0045] The setting unit 221 detects, from the input language, a word that indicates a spatial position in the input image, and sets an important region based on the detection result. Examples of the word include "foreground," "back," and "close by."

[0046] For example, when the setting unit 221 detects "foreground" or "close to hand" from the input language, it sets an area that includes at least a portion of an area below the center of the input image as the important area. For example, when the setting unit 221 detects "back" from the input language, it sets an area that includes at least a portion of an area above the center of the input image as the important area. Generally, in an image captured by the imaging unit 24, an image of an object that is closer to the user is included in the lower part of the image, and an image of an object that is further away from the user is included in the upper part of the image. Therefore, by setting the important area as described above, it is possible to include the image of the target of attention in the important area. Note that the shape and size of the area corresponding to the detected language can be adjusted as desired.

[0047] Furthermore, when capturing an input image, the capturing unit 24 can also acquire rotation information of the input image. In this case, the transmitting device 2 may include, for example, an acceleration sensor. The rotation information may be, for example, the angle of the upper or lower side of the capturing unit 24 relative to the horizontal plane. If the angle indicated by the rotation information is not 0°, the setting unit 221 rotates (corrects) the input image so that the angle becomes 0°, and then sets the important area.

[0048] Also, consider a case where the setting unit 221 detects, from the input language, a language indicating a position in the input image, along with a reference language that serves as a reference for that position. In this case, the setting unit 221 may specify, in the input image, the position indicated by the above language as a relative position from an image of an object corresponding to the reference indicated by the above reference language, and set an important region. For example, if the setting unit 221 detects the language "in front of the tree" from the input language, it specifies "tree" as the reference language (position reference). Then, the setting unit 221 detects an image of a "tree" in the input image, and sets an area adjacent to and below the image of the "tree" as an important region.

[0049] (Setting Example 5) The setting unit 221 sets an important region by using language (target designation information) included in the input language for specifying what the target of attention is.

[0050] The setting unit 221 detects, for example, language indicating the name of an object (e.g., apple, orange, etc.) from the input language and sets an important region based on the detection result. The setting unit 221 identifies the image of the object indicated by the language as the image of the target of attention and sets an area including the image as the important region. The setting unit 221 sets the important region including the image of the target of attention by searching the input image for an area corresponding to the detected language using, for example, known image recognition technology.

[0051] Furthermore, the setting unit 221 detects an image of an object from the input image using, for example, a known image recognition technique, then identifies the name of the detected object and acquires information associating the identified name of the object with the position of the image of the object in the input image. Then, when the setting unit 221 determines that the input image includes an image of an object having a name that matches a language included in the input language, it sets a region in the input image that includes the image of the object corresponding to the name as an important region.

[0052] Consider a case where the input language includes a language indicating an object of interest (equivalent to "bird"), such as "a bird sitting in a tree," and a language limiting the object of interest (equivalent to "tree"). In this case, the setting unit 221 sets an area including an image of an object indicated by the language limiting the object of interest (an area including an image of a "tree") as a candidate area, and then sets an area including an image of the object of interest included in the candidate area (an area including an image of a "bird") as an important area. The setting unit 221 can determine whether the input language includes a language indicating an object of interest and a language limiting the object of interest by, for example, using a known text analysis technique.

[0053] (Setting Example 6) The setting unit 221 sets an important region by combining at least two or more of the setting methods of setting examples 1 to 5. The setting unit 221 sets the result (intersection) of integrating the important regions individually set from setting examples 1 to 5 as the final important region. For example, in response to the input language "What's that thing on the right in the foreground?", the setting unit 221 integrates an important region set based on "foreground" and an important region set based on "right" to set the final important region. In this example, the setting unit 221 sets the lower right region when the input image is divided into four as the important region.

[0054] Furthermore, when setting example 5 is applied as one of the setting methods, the setting unit 221 can set the final important region, for example, as follows. The setting unit 221 first sets, in the input image, a region including the image of the target of interest indicated by the target designation information as a candidate region. When the input image includes multiple images of the target of interest, the setting unit 221 identifies, as the candidate region, a region including the images of all of the targets of interest. For example, when an input language such as "What is the bird above?" and an input image including images of multiple birds are acquired, the setting unit 221 sets, as the candidate region, a rectangular region including the images of all of the birds. Then, in specifying a position using the image position designation information, the setting unit 221 regards the candidate region as the input image, and sets the "upper" region of the candidate region (for example, the region above the center of the candidate region) as the important region.

[0055] The important areas set by the methods shown in the above setting examples 1 to 5 (individual setting examples) are set as areas smaller than the input image, so the amount of data (transmission amount) sent to the response generation unit 311 can be further reduced. This allows the response generation unit 311 to further reduce the time it takes to generate a response language. Furthermore, because the important areas can be set more accurately, the accuracy (response quality) of the response language generated by the response generation unit 311 can be improved.

[0056] Furthermore, since the important area set by the method shown in setting example 6 above is set as an area smaller than the important area set using the individual setting examples, the amount of data (transmission amount) sent to the response generation unit 311 can be further reduced. Therefore, the time it takes for the response generation unit 311 to generate a response language can be further shortened. Furthermore, since the important area can be set more accurately, the accuracy (response quality) of the response language by the response generation unit 311 can be improved.

[0057] Next, the first adjustment unit 222 adjusts the input image acquired in S1 based on the important area set in S2 to generate a first adjusted image (S3; first adjustment step). As shown in FIG. 2 , the first adjustment unit 222 transmits the generated first adjusted image to the transmission control unit 223.

[0058] The first adjustment unit 222 generates the first adjusted image using, for example, any one of the methods in the following adjustment examples 1 to 4, or using at least two or more of the methods in the adjustment examples 1 to 4. The processing of the first adjustment unit 222 makes it possible to generate a first adjusted image in which the amount of data of the input image is reduced.

[0059] (Adjustment Example 1) The first adjustment unit 222 crops an important area from an input image and generates the cropped image as a first adjusted image. This allows the first adjustment unit 222 to generate a first adjusted image with a significantly reduced amount of data from the input image. Furthermore, the first adjustment unit 222 can generate the first adjusted image through relatively lightweight processing.

[0060] (Adjustment Example 2) The first adjustment unit 222 generates a first adjusted image by performing image processing to reduce the amount of data in non-important regions other than the important region in the input image. For example, the image processing can be a process of filling the non-important region with a single color (a process of setting a single gradation value). Other examples of the image processing can include a color reduction process for the non-important region, a brightness range restriction process, or a process using a low-pass filter. When a low-pass filter is used, the first adjustment unit 222 can apply the low-pass filter to the non-important region to reduce high-frequency components in the non-important region, thereby reducing the amount of data in the non-important region. For example, a Gaussian filter can be used as the low-pass filter.

[0061] By performing the above processing, the first adjustment unit 222 can generate a first adjusted image in which the data amount of the input image is significantly reduced. Note that in the case of Adjustment Example 2, the amount of reduced data is less than in Adjustment Example 1, but the first adjusted image includes non-important areas. Therefore, when Adjustment Example 2 is applied, the data amount is reduced compared to the input image, while enabling the response generation unit 311 to utilize information regarding which parts of the input image are important areas.

[0062] (Adjustment Example 3) The first adjustment unit 222 generates a first adjusted image by performing lossy compression processing on the input image so that the image quality of the important region is improved compared to the image quality of the non-important region. When an image is transmitted via the network 4, lossy compression processing, such as JPEG (Joint Photographic Experts Group) is generally performed. Adjustment Example 3 utilizes this technology. Specifically, the first adjustment unit 222 compresses the non-important region more strongly than the important region (the compression rate of the non-important region is set higher than the compression rate of the important region). When quantization technology is used for compression processing, the first adjustment unit 222 can achieve such compression processing by making the quantization width (quantization parameter) of the important region smaller than the quantization width of the non-important region.

[0063] (Adjustment Example 4) The first adjustment unit 222 generates data in which the important region is represented in high resolution and the entire input image is represented in low resolution as the first adjusted image. In this case, the first adjustment unit 222 does not generate a single image as the first adjusted image, but generates two images, a high-resolution image of the important region and a low-resolution image of the entire input image, as the first adjusted image.

[0064] For example, the first adjustment unit 222 applies adjustment example 1 to the input image to generate an image obtained by cropping an important region from the input image as a high-resolution image of the important region. Furthermore, the first adjustment unit 222 generates an image obtained by downsampling the input image to one-fourth both vertically and horizontally as a low-resolution image of the entire input image. The amount of downsampling can be adjusted as desired.

[0065] According to adjustment example 4, the response generation unit 311 can generate a response language for an important region by taking into account a non-important region while reducing the amount of data compared to the input image. Even if the important region set for the input language is set so as to be shifted from the region that should originally be of interest, the response generation unit 311 can generate a response language that is estimated to be correct from the input language and the non-important region. Furthermore, when generating a response language for an important region, the response generation unit 311 can also generate a response language that takes into account the content of the image (e.g., an image of an object) reflected in the non-important region.

[0066] In this way, the setting unit 221 can set the important region by applying at least one setting method from among a plurality of setting methods. The first adjustment unit 222 can apply at least one adjustment method from among a plurality of adjustment methods to generate a first adjusted image based on the set important region. Therefore, the data amount of the input image can be reduced depending on the scene represented by the input image, the content of the input language, or the purpose of the processing. Note that examples of the purpose of the processing include performing processing that prioritizes reducing the data amount, or performing processing that prioritizes the accuracy of the response language.

[0067] Next, the transmission control unit 223 generates MM instruction data (instruction data) including the input language acquired in S1 and the first adjustment image generated in S3. Then, the transmission control unit 223 transmits the generated MM instruction data to the response generation unit 311 (S4).

[0068] The transmission control unit 223 may add, for example, area specifying information that specifies an important area in the first adjustment image to the MM instruction data.

[0069] When the first adjustment image includes a non-important region, the transmission control unit 223 may add information indicating which portion of the first adjustment image corresponds to the important region as region identification information to the MM instruction data. In this case, the transmission control unit 223 adds information specifying the position of the important region in the first adjustment image as region identification information to the MM instruction data. Furthermore, when multiple images are generated as the first adjustment image (when the first adjustment image is composed of multiple images), the transmission control unit 223 may add information indicating which image corresponds to the important region as region identification information to the MM instruction data.

[0070] Furthermore, the transmission control unit 223 may add, for example, generation type information indicating the generation type of the first adjustment image to the MM instruction data.

[0071] The transmission control unit 223 may add, as generation type information, information indicating which of the important region setting methods the setting unit 221 has applied to the MM instruction data. In the above example, the transmission control unit 223 may add, as generation type information, information indicating which of setting examples 1 to 6 has been applied to the MM instruction data. The transmission control unit 223 may also add, as generation type information, information indicating which of the first adjustment image generation methods (adjustment methods) the first adjustment unit 222 has applied to the MM instruction data. In the above example, the transmission control unit 223 may add, as generation type information, information indicating which of adjustment examples 1 to 4 has been applied to the MM instruction data.

[0072] By including the region specifying information and / or the generation type information in the MM instruction data, the response generating unit 311 can also use this information to interpret the received first adjusted image and generate a response language.

[0073] Next, in the receiving device 3, the response generation unit 311 receives the MM instruction data transmitted in S4. Then, the response generation unit 311 uses the first language model 321 to generate a response language (response) for the combination of the input language acquired in S1 and the first adjustment image generated in S3 (S5; response generation step). The response generation unit 311 transmits data indicating the generated response language to the presentation control unit 213 of the transmitting device 2 (S6).

[0074] Next, in the transmitting device 2, the presentation control unit 213 receives the data indicating the response language transmitted in S6. The presentation control unit 213 transmits the received data indicating the response language and an instruction to present the data to the presentation unit 23. As a result, the presentation unit 23 presents the response language for the input language and input image acquired in S1 (S7).

[0075] <Other Processing> The first control unit 21 may perform the following processing, for example.

[0076] (Important Region Presentation Process) The presentation control unit 213 may present the important region set by the setting unit 221 via the presentation unit 23 .

[0077] For example, when the transmission control unit 223 transmits MM instruction data to the response generation unit 311, the presentation control unit 213 presents area setting information indicating which area of ​​the input image has been set as an important area via the presentation unit 23. This allows the user to understand what kind of image is the premise for the response language generated by the response generation unit 311. This allows the user to more deeply understand the content of the response language.

[0078] It is preferable that the presentation control unit 213 presents the region setting information at an earlier timing than the presentation of the response language of the response generation unit 311. This allows the user to estimate the expected response language at a relatively earlier timing, thereby reducing the possibility that the user will be stressed by waiting for the generation of a response language.

[0079] The presentation control unit 213 may present the important region itself as the region setting information, or may present information that enables the user to understand the important region (information corresponding to the important region) instead of the important region itself. As the information, the presentation control unit 213 may present, for example, an image in which the important region is cut out, or may present a frame that enables the user to identify the important region in the input image. The information corresponding to the important region does not necessarily have to be an image, but may also be in another form. For example, the information may be presented by displaying or playing as audio the coordinates indicating the position of the important region in the input image, or the size of the important region expressed in language.

[0080] When the presentation control unit 213 presents the area setting information at an earlier timing than the presentation of the response language by the response generation unit 311, the transmission data adjustment unit 212 may have a function to cancel the generation of the response language by the response generation unit 311. For example, when the transmission data adjustment unit 212 acquires a user instruction to cancel the generation of the response language by the response generation unit 311, the transmission data adjustment unit 212 transmits an instruction to cancel the generation of the response language to the response generation unit 311. When the response generation unit 311 receives the cancellation instruction, it stops the generation process of the response language.

[0081] This eliminates the need for the user to wait for the generation of a response language different from the user's expected response language (a response language undesirable to the user) when an important region (an important region unintended by the user) is set in a position in the input image different from the position expected by the user. Therefore, the user can promptly have the response generation unit 311 re-generate the response language without waiting for the generation of an undesirable response language. Specifically, the user can input a response language re-generation instruction to have the transmission data adjustment unit 212 re-set the important region and re-generate the first adjustment image, and have the response generation unit 311 re-generate the response language based on the re-generated first adjustment image. This reduces the likelihood of the user experiencing stress due to the generation of a response language undesirable to the user.

[0082] (Processing for requesting transmission of important area) The transmission control unit 223 may transmit MM instruction data to the response generation unit 311 when permission to transmit information indicating the presented important area to the response generation unit 311 is obtained from the user.

[0083] The transmission data adjustment unit 212 may have a function of determining whether or not the transmission control unit 223 is permitted to transmit MM instruction data when the presentation control unit 213 presents area setting information as described above. For example, after the user checks the area setting information presented by the presentation unit 23, the user inputs a user instruction to permit transmission of information indicating an important area (transmission of MM instruction data in this embodiment) via the input unit 22. When the transmission data adjustment unit 212 acquires the user instruction, it permits the transmission control unit 223 to transmit the MM instruction data to the response generation unit 311.

[0084] This reduces the possibility that a response language that is undesirable to the user will be generated when an important area that the user did not intend is set, thereby reducing the possibility that the user will be stressed by the generation of such a response language.

[0085] Furthermore, by presenting the region setting information by the presentation control unit 213, when an input image contains a scene that may contain personal information, the user can confirm whether the set important region poses any security or privacy issues. As described above, the transmission data adjustment unit 212 transmits the MM instruction data to the response generation unit 311 only when permission to transmit the MM instruction data is obtained. In other words, if the user determines that the set important region poses security or privacy issues, the transmission control unit 223 does not transmit the MM instruction data. This makes it possible to improve security and privacy in the response language generation process.

[0086] (Important Region Correction Process) After the presentation control unit 213 presents the region setting information, the transmission data adjustment unit 212 may acquire a user correction instruction (additional input language) to correct the important region set by the setting unit 221. The input unit 22 may, for example, accept the user's correction instruction verbally or as text information. When the transmission data adjustment unit 212 acquires the user's correction instruction, the setting unit 221 re-sets the important region in the input image based on the user's correction instruction. The first adjustment unit 222 re-generates a first adjustment image based on the important region re-set by the setting unit 221. The transmission control unit 223 includes the first adjustment image re-generated by the first adjustment unit 222 in MM instruction data and transmits the MM instruction data to the response generation unit 311. Note that the user's correction instruction is not necessarily used only for resetting the important region once, but may also be used for setting an important region in a subsequent inquiry in another input language.

[0087] The user's correction instruction may be, for example, a position adjustment instruction to adjust the position of the important region. The position adjustment instruction may be language including a word or phrase indicating a direction, such as "a little further to the right" or "lower." The setting unit 221 moves the position of the important region that has already been set in the input image based on the word or phrase indicating a direction included in the position adjustment instruction, and sets a new important region.

[0088] Furthermore, the user's correction instruction may be, for example, a size adjustment instruction for adjusting the size of the important area. Examples of size adjustment instructions include a first setting instruction requesting a wider range setting and a second setting instruction requesting a narrower range setting. Examples of the first setting instruction include an instruction such as "check the periphery more." Examples of the second setting instruction include an instruction such as "focus on the center."

[0089] When the transmission data adjustment unit 212 receives a first setting instruction, the setting unit 221 sets an area obtained by enlarging the size of an important area that has already been set as a new important area. On the other hand, when the transmission data adjustment unit 212 receives a second setting instruction, the setting unit 221 sets an area obtained by reducing the size of an important area that has already been set as a new important area.

[0090] Furthermore, the user's correction instruction may be, for example, a target change instruction to change the target of attention in the important region. The target change instruction can also be described as an instruction to select an image of a target of attention that is different from the image of the target of attention included in the important region set by the setting unit 221 when multiple images of targets of attention included in the input language are present in the input image. For example, consider a case where the input image shows two dogs and the setting unit 221 sets a region including the image of one of the dogs as the important region. In this case, an example of a target change instruction is an instruction such as "the one other than that one." When such a target change instruction is acquired, the setting unit 221 selects an image of a dog other than the image of the dog included in the important region that has already been set, and sets the region including the image of the other dog as a new important region.

[0091] The setting unit 221 can re-set important areas in the input image based on the acquired user correction instructions. This allows the response generation unit 311 to generate a response language for the target of attention that is more in line with the user's intention. Note that the setting unit 221 can interpret the content of the user correction instructions by, for example, using a known text analysis technique.

[0092] Furthermore, the transmission data adjustment unit 212 may repeat obtaining a user's correction instruction and presenting the reset important area until permission from the user is obtained. The reset important area may also be presented as information corresponding to the important area, rather than the important area itself.

[0093] (Processing for Changing the Setting Method of Important Regions According to Communication Status) The setting unit 221 may adjust the setting range of important regions based on the communication status with the receiving device 3 .

[0094] The transmission data adjustment unit 212 acquires data indicating the state of the transmission path between the transmitting device 2 and the receiving device 3, which is the destination of the MM instruction data. The setting unit 221 adjusts the setting range of the important region by selecting a setting method for the important region based on the acquired state of the transmission path. If the transmission path is realized by a network 4, the state of the transmission path may be the state of the communication bandwidth of the network 4. The state of the communication bandwidth may be, for example, at least one of the effective bandwidth, maximum bandwidth, minimum bandwidth, and estimated bandwidth. The state of the communication bandwidth may be specified, for example, by the capacity of the transmission path.

[0095] The transmission data adjustment unit 212 determines whether the capacity of the transmission path is sufficient to transmit the MM instruction data. For example, the transmission data adjustment unit 212 determines that the capacity of the transmission path is sufficient (i.e., the transmission path is wide) when the capacity of the transmission path is equal to or greater than a preset threshold. Furthermore, the transmission data adjustment unit 212 may determine that the capacity of the transmission path is sufficient when, for example, the time required to transmit the amount of MM instruction data generated by the transmission control unit 223 is within a preset threshold, assuming the capacity of the transmission path.

[0096] If the transmission data adjustment unit 212 determines that the capacity of the transmission path is sufficient, the transmission control unit 223 transmits MM instruction data including the important region set by the setting unit 221 to the response generation unit 311. On the other hand, if the transmission data adjustment unit 212 determines that the capacity of the transmission path is insufficient, the setting unit 221 selects a setting method that allows for setting an important region that is smaller than the setting method that would be selected if the capacity of the transmission path were sufficient. The setting unit 221 then sets an important region for the input image using the selected setting method. For example, if the network 4 is congested and the communication bandwidth is relatively narrow, the transmission data adjustment unit 212 determines that the capacity of the transmission path is insufficient, and the setting unit 221 selects a setting method that allows for setting an important region that is smaller than the setting method that would be selected if the network 4 were not congested.

[0097] As a result, even if the transmission path from the transmission data adjustment unit 212 to the response generation unit 311 is narrow, the setting unit 221 sets the important region to be small, which allows the first adjustment unit 222 to generate a first adjustment image with a smaller amount of data. This also reduces the amount of MM instruction data generated by the transmission control unit 223. When large data is transmitted over a narrow transmission path, delays in data transmission increase, but by reducing the amount of MM instruction data, the possibility of delays in data transmission can be reduced.

[0098] (Processing for Changing the Method of Setting Important Regions According to Performance or Usage Status of First Language Model) The setting unit 221 may adjust the setting range of important regions based on the usage status of the first language model 321 .

[0099] The transmission data adjustment unit 212 acquires, for example, data indicating the usage state (computational resources) of the response generation unit 311 from the response generation unit 311 as the usage state of the first language model 321. As the usage state of the response generation unit 311, for example, at least one of the usage rate of the CPU (Central Processing Unit), the usage rate of the GPU (Graphics Processing Unit), the memory usage rate, and the video memory usage rate of the second control unit 31 can be used.

[0100] The transmission data adjustment unit 212 determines whether the usage rate of the second control unit 31 is less than a predetermined threshold. If the transmission data adjustment unit 212 determines that the usage rate of the second control unit 31 is less than the predetermined threshold, the transmission control unit 223 determines that the usage rate is low and transmits MM instruction data including the important area set by the setting unit 221 to the response generation unit 311. On the other hand, if the transmission data adjustment unit 212 determines that the usage rate of the second control unit 31 is equal to or greater than the predetermined threshold, the setting unit 221 selects a setting method that can set a smaller important area than the setting method selected when the usage rate is low. Then, the setting unit 221 sets the important area for the input image using the selected setting method.

[0101] Furthermore, the setting unit 221 may adjust the setting range of the important region based on the performance of the first language model 321 .

[0102] The transmission data adjustment unit 212 acquires data indicating the performance of the first language model 321 from the response generation unit 311. As the performance of the first language model 321, for example, data indicating the size of the first language model 321 is acquired. Examples of the size of the first language model 321 include the number of parameters of the neural network that constitutes the first language model 321 and the memory size required for processing using the first language model 321.

[0103] The transmission data adjustment unit 212 determines whether the size of the first language model 321 is equal to or greater than a predetermined threshold. If the transmission data adjustment unit 212 determines that the size of the first language model 321 is equal to or greater than the predetermined threshold, the transmission control unit 223 regards the size as large and transmits MM instruction data including the important region set by the setting unit 221 to the response generation unit 311. On the other hand, if the transmission data adjustment unit 212 determines that the size of the first language model 321 is less than the predetermined threshold, the setting unit 221 selects a setting method that can set an important region that is smaller than the setting method selected when the size is large. Then, the setting unit 221 sets an important region for the input image using the selected setting method.

[0104] As a result, when the processing performance of the response generation unit 311 is low, for example, when the response generation unit 311 has few free computational resources or when the processing performance of the first language model 321 is low, the setting unit 221 can set a small important region that takes this processing performance into consideration. This allows the amount of data in the first adjustment image generated by the first adjustment unit 222 to be reduced. When the processing performance of the response generation unit 311 is low and a first adjustment image with a large amount of data is input to the response generation unit 311 to generate a response language, it takes time to generate the response language, increasing the delay in presenting the response language. On the other hand, by reducing the amount of data in the first adjustment image, it is possible to reduce the possibility of an increase in the time required to generate the response language. This reduces the possibility of a delay in presenting the response language.

[0105] (Processing for Setting Important Regions Based on Shooting Conditions) The setting unit 221 may set important regions based on the shooting conditions of the input image in addition to the input language.

[0106] The shooting state may indicate, for example, whether handheld shooting is being performed (handheld shooting state). The handheld shooting state indicates whether a user is holding a camera or a camera-equipped device, such as a smartphone, in their hand to capture an input image. In this embodiment, the handheld shooting state indicates whether a user is holding the transmission device 2, which includes the shooting unit 24, in their hand to capture an input image. The handheld shooting state can be estimated, for example, from the detection results over time of a position sensor (not shown) or an acceleration sensor (not shown) included in the transmission device 2, or from changes in the input image acquired over time.

[0107] The transmission data adjustment unit 212 acquires the detection results of the position sensor or acceleration sensor from the position sensor or acceleration sensor as data indicating the shooting state. Alternatively, the transmission data adjustment unit 212 can acquire changes over time in the input image using known image analysis techniques.

[0108] The transmission data adjustment unit 212 determines the handheld shooting state based on data indicating the shooting state. If the transmission data adjustment unit 212 determines that handheld shooting is being performed, the setting unit 221 sets an important region for the input image so that an area including the center of the input image is given priority in being included in the important region. If handheld shooting is being performed, the user can adjust the shooting direction of the shooting unit 24. Therefore, there is a high possibility that the image of the target included in the input language will be included in the center of the input image. Therefore, if it is determined that handheld shooting is being performed, the setting unit 221 sets an important region by giving priority to an area including the center of the input image, thereby enabling a more appropriate important region to be set for the input image.

[0109] On the other hand, when the transmission data adjustment unit 212 determines that handheld shooting is not being performed, the setting unit 221 does not need to set an important region for the input image so that an area including the center of the input image is preferentially included in the important region. This is because when handheld shooting is not being performed, that is, when the transmission device 2 (photographing unit 24) is installed, the direction of the photographing unit 24 is not necessarily adjusted so that the image of the target included in the input language is included in the center of the input image.

[0110] The shooting state may also indicate, for example, whether the input image is fixed at a specific position (fixed-point shooting state). The fixed-point shooting state can be estimated from, for example, the detection results over time of a position sensor (not shown) or an acceleration sensor (not shown) included in the transmitting device 2.

[0111] The transmission data adjustment unit 212 acquires the detection result of the position sensor or acceleration sensor from the position sensor or acceleration sensor as data indicating the shooting state, and determines the fixed point shooting state based on the data indicating the shooting state.

[0112] When the transmission data adjustment unit 212 determines that the image is in a fixed-point shooting state, the setting unit 221 sets an important region for the input image so as to prioritize areas that have changed in the input image acquired over time and include them in the important region. In the case of fixed-point shooting, there is a high possibility that the user will be interested in changed parts of the input image, such as when the image of an object included in the input image moves, when the image of a new object appears in the input image, or when the image of an object disappears from the input image. In addition, users often make inquiries about the changed parts. Therefore, by having the setting unit 221 prioritize such parts in setting the important region, the response generation unit 311 can generate a higher quality response language.

[0113] The shooting state may also indicate, for example, whether the input image has been shot with a wide-angle camera (wide-angle shooting state). The transmission data adjustment unit 212 acquires, as data indicating the shooting state, information indicating whether the shooting unit 24 is a wide-angle camera, from the first storage unit 25.

[0114] If the transmission data adjustment unit 212 determines that the input image was captured with a wide angle (i.e., if the image capture unit 24 is determined to be a wide-angle camera), it sets an important area that is smaller than the important area set for an input image that was not captured with a wide angle. In the case of wide-angle capture, the proportion of the area in the input image where the image of the target object included in the input language is located is often relatively small. Therefore, in the case of wide-angle capture, even if the setting unit 221 sets a relatively small important area, there is a high possibility that the image of the target object can be contained within the important area. In this case, the data amount of the first adjustment image can be reduced, and therefore the data amount of the MM instruction data can also be reduced.

[0115] (Processing for Setting Important Regions for Video) The acquisition unit 211 may acquire a video having multiple frames as an input image. In this case, the setting unit 221 sets an important region by regarding the frame corresponding to the time when the input language is input as the input image. This allows the response generation unit 311 to generate a response language even for input images that are captured continuously over time. Video has a larger amount of data than images. Therefore, when the input image is a video, it is particularly useful for the setting unit 221 to set important regions for the above frames.

[0116] The time at which the input language is input may be any time between the time at which the input of the input language is started and the time at which the input of the input language is finished.

[0117] Furthermore, when the input image is a video, the transmission data adjustment unit 212 does not need to generate first adjustment images in which important regions are set for all input languages. When a predetermined condition is satisfied, the transmission data adjustment unit 212 may omit the process of setting important regions and the process of generating first adjustment images, and may transmit the input language and an instruction to reuse the already transmitted first adjustment images in MM instruction data to the response generation unit 311.

[0118] The predetermined condition may be, for example, the degree of similarity between the important region set by the setting unit 221 when the input language is acquired and an important region previously set by the setting unit 221. As the degree of similarity, for example, a distance index calculated from the difference between corresponding pixels, such as the L1 distance between images indicating the important region or PSNR (Peak Signal-to-Noise Ratio), may be used.

[0119] If the transmission data adjustment unit 212 determines that the similarity is less than a preset threshold (if the similarity is determined to be low), it performs the following process. As described above, the setting unit 221 sets an important area for the input image based on the acquired input language, and the first adjustment unit 222 generates a first adjusted image based on the important area. Then, the transmission control unit 223 transmits MM instruction data including the acquired input language and the generated first adjusted image to the response generation unit 311.

[0120] On the other hand, if the transmission data adjustment unit 212 determines that the similarity is equal to or greater than a preset threshold (if the similarity is determined to be high), it performs the following process: The transmission control unit 223 includes an instruction to reuse the first adjustment image generated based on the previously set important area in the MM instruction data together with the acquired input language, and transmits the MM instruction data to the response generation unit 311. That is, in this case, the transmission data adjustment unit 212 does not set the important area or generate the first adjustment image as described above, and does not transmit the first adjustment image.

[0121] The degree of similarity may be, for example, the size of the important area or the degree of change in its position in the input image. In this case, the transmission data adjustment unit 212 may determine that the degree of change is high when it determines that the degree of change is less than a preset threshold (when it determines that the degree of change is small).

[0122] In this way, by utilizing the continuity of the image, the data amount (transmission amount) of the MM instruction data can be reduced. In particular, when reusing a past first adjustment image, the image data amount in the MM instruction data can be reduced to zero.

[0123] In the above example, if the transmission data adjustment unit 212 determines that the similarity is high, the transmission data adjustment unit 212 does not transmit the first adjustment image, but may transmit a first adjustment image that has been adjusted to have a smaller data amount than when the similarity is determined to be low. In this case, for example, the transmission control unit 223 transmits the first adjustment image generated by the first adjustment unit 222 as is to the response generation unit 311. On the other hand, for first adjustment images generated subsequently by the first adjustment unit 222, the transmission control unit 223 compresses the first adjustment image to reduce its resolution and then transmits it to the response generation unit 311. In this case, as in the above, the data amount (transmission amount) of the MM instruction data can be reduced by utilizing the continuity of the video.

[0124] (Process for setting important regions by acquiring related languages ​​in advance) The acquisition unit 211 may acquire related languages ​​related to the target of attention indicated by the input language before the input language. In this case, the setting unit 221 sets important regions using the related languages. The related languages ​​are languages ​​that are commonly applied to all input languages.

[0125] The related language may be, for example, a pre-target attention instruction such as the name of an object common to the multiple input images (a target object that the first language model 321 should focus on in common in the multiple input images), a target classification, and target attributes.

[0126] Attributes of an object include, for example, the color of the object, the state of movement of the object, and the size of the object. The color of the object includes, for example, colors such as red and white, as well as states such as bright, shining, and blinking. The state of movement of the object includes, for example, states such as stationary, moving, running, and flying. The size of the object includes, for example, large, small, thick, thin, thick, thin, long, and short.

[0127] When the acquisition unit 211 acquires a pre-target attention instruction as a related language, the setting unit 221 narrows down the area of ​​the input image that corresponds to the related language. Next, the setting unit 221 regards the area narrowed down based on the related language as the input image and sets an important area that corresponds to the input language acquired by the acquisition unit 211. This allows the first adjustment unit 222 to generate a first adjusted image based on the important area that is set in consideration of not only the input language but also the related language.

[0128] For example, suppose that the user first inputs a related phrase such as "I want to enjoy watching animals today" into the input unit 22. Then, the user spots a squirrel in a tree in the distance, takes a photograph of the squirrel, and uses the photographed image as an input image. Then, the user inputs the input phrase "What is that?" into the input unit 22, thereby requesting the response generation unit 311 to generate a response phrase.

[0129] In this case, the setting unit 221 selects an area where "animals" exist from the input image based on the related language and identifies the selected area as a candidate area for the important area. Next, the setting unit 221 selects an area for an animal that is far away from the candidate areas for the important area based on the input language and sets the selected area as the important area.

[0130] When no related language is input, objects other than animals (such as "trees") may also be the focus of attention, and therefore the setting unit 221 may set a broad area including images of multiple objects as the important area. On the other hand, when related language is input in advance, the setting unit 221 can narrow down the area using the related language before setting the important area. This allows the setting unit 221 to set a small important area. This enables the response generation unit 311 to generate a response language with high accuracy while reducing the amount of data in the MM instruction data.

[0131] The related language may be, for example, a pre-ignore target instruction such as a name of an object, a classification of an object, or an attribute of an object that should not be paid attention to in common in a plurality of input images by the first language model 321. In this case, the setting unit 221 sets an area that does not include an object specified in the pre-ignore target instruction as a candidate area for an important area.

[0132] <Effects of the Response System> As described above, according to the response system 1, the transmission data adjustment unit 212 sets an important area in the input image based on the input language and generates a first adjustment image so as to preferentially include the important area. Then, the response generation unit 311 uses the first language model to generate a response language for the combination of the input language and the first adjustment image. This increases the likelihood that the response generation unit 311 will generate a response language that meets the user's expectations. In other words, the quality of the response language generated by the response generation unit 311 can be improved. Furthermore, by generating the first adjustment image so as to preferentially include the important area, it is possible to generate a first adjustment image with a smaller amount of data than the input image. This reduces the processing load and the likelihood of processing delays occurring in the response generation unit 311.

[0133] Furthermore, by generating the first adjustment image as described above, the amount of MM instruction data can be reduced. This reduces the possibility of increasing the communication bandwidth of the network 4. Even if the communication bandwidth is narrow, the possibility of delays occurring due to the transmission of the MM instruction data can be reduced, and the MM instruction data can be transmitted in a short time. This makes it possible to generate a response language in a short time or with little delay.

[0134] [Embodiment 2] Another embodiment of the present disclosure will be described below. For ease of explanation, the same reference numerals will be used to designate components having the same functions as those described in the above embodiment, and the description thereof will not be repeated. The same applies to other embodiments.

[0135] Fig. 4 is a block diagram showing an example of the configuration of a response system 1A of embodiment 2. As shown in Fig. 4, the response system 1A differs from the response system 1 of embodiment 1 in that it includes a receiving device 3A instead of the receiving device 3. The receiving device 3A differs from the receiving device 3 of embodiment 1 in that it includes a second control device 31A instead of the second control device 31. The second control device 31A differs from the second control device 31 of embodiment 1 in that it includes a second adjustment unit 312. Also, Fig. 4 omits the configuration of the transmitting device 2 shown in Fig. 1.

[0136] <Processing Flow> An example of the processing flow in the response system 1A will be described with reference to Fig. 4 and Fig. 5. Fig. 5 is a sequence diagram showing an example of the processing flow in the response system 1A. In Fig. 5, the same processing as in Fig. 3 will not be described.

[0137] 5 , the transmission control unit 223 transmits MM instruction data including at least the input language and the first adjustment image to the second adjustment unit 312 (S4). The second adjustment unit 312 generates a second adjustment image by further adjusting the received first adjustment image based on the received input language (S11). The second adjustment unit 312 transmits data including the received input language and the generated second adjustment image to the response generation unit 311 as new MM instruction data.

[0138] The second adjustment unit 312 further adjusts the first adjusted image to generate a second adjusted image so as to more accurately include the target of interest indicated by the input language. Specifically, the second adjustment unit 312 sets an area including the target of interest based on the object designation information included in the input language, and performs processing to cut out the area from the first adjusted image. The setting of the area uses the processing of Setting Example 5 of the setting unit 221 described above.

[0139] For example, for an input language such as "Tell me about the dog in the middle?", the setting unit 221 performs the processing of setting example 3. Specifically, the setting unit 221 sets the area corresponding to "middle" in the input image as the important area based on image position designation information included in the input language. After the first adjustment unit 222 generates the first adjusted image based on the important area, the transmission control unit 223 transmits MM instruction data including the input language and the first adjusted image to the second adjustment unit 312. The second adjustment unit 312 identifies "dog" included in the received input language as the target of attention, and performs image analysis processing on the first adjusted image to perform processing to cut out the area including "dog" from the first adjusted image. As a result, the second adjustment unit 312 can generate a second adjusted image in which the area including the cut-out "dog" is reset as the important area.

[0140] After the process of S11, the response generation unit 311 uses the first language model 321 to generate a response language for data corresponding to a combination of the input language received from the transmission control unit 223 and the first adjusted image generated by the first adjustment unit 222 (S12). The response generation unit 311 generates a response language for the combination of the input language and the second adjusted image received from the second adjustment unit 312 as the data. The combination of the input language and the second adjusted image is an example of data corresponding to a combination of the input language and the first adjusted image.

[0141] In this way, the second adjustment unit 312 can more accurately extract an area containing the image of the target object from the first adjustment image to generate the second adjustment image, thereby improving the quality of the response language compared to when the first adjustment image is input to the response generation unit 311.

[0142] In general, in response systems, the receiving device is often implemented as a cloud server or the like. In this case, the processing of the second control unit of the receiving device is performed in an environment with relatively abundant computational resources. On the other hand, the transmitting device is often implemented as a mobile terminal or the like. In this case, the processing of the first control unit of the transmitting device is performed in an environment with relatively scarce computational resources. Therefore, the setting unit 221 sets a broad important area, and the second adjustment unit 312 further narrows the important area using text analysis processing and image analysis processing, thereby enabling more accurate setting of the important area while taking into account the computational resources of the transmitting device 2 and the receiving device 3A.

[0143] [Third embodiment] Fig. 6 is a block diagram showing an example of the configuration of a response system 1B of a third embodiment. Fig. 7 is a block diagram showing an example of a transmission data adjustment unit 212B of the third embodiment.

[0144] As shown in FIG. 6 , the response system 1B differs from the response system 1A of the second embodiment in that it includes a transmitting device 2B instead of the transmitting device 2 and a receiving device 3B instead of the receiving device 3A. The transmitting device 2B differs from the transmitting device 2 of the first and second embodiments in that it includes a first control unit 21B instead of the first control unit 21. As shown in FIG. 7 , the first control unit 21B includes a transmission data adjusting unit 212B including a request receiving unit 224 in addition to a setting unit 221, a first adjusting unit 222, and a transmission control unit 223. Also, as shown in FIG. 6 , the receiving device 3B differs from the receiving device 3A of the second embodiment in that it includes a second control unit 31B instead of the second control unit 31A. The second control unit 31B includes a request unit 313 in addition to a response generating unit 311 and a second adjusting unit 312.

[0145] <Processing Flow> An example of the processing flow in the response system 1B will be described using Figures 6 to 8. Figure 8 is a sequence diagram showing an example of the processing flow in the response system 1B. In Figure 8, the same processing as in Figure 5 will not be described.

[0146] As shown in FIG. 8, the transmission control unit 223 transmits MM instruction data including at least the input language and the first adjustment image to the second adjustment unit 312 and the request unit 313 (S4).

[0147] The second adjustment unit 312 generates a second adjusted image by further adjusting the received first adjusted image based on the received input language (S11). The request unit 313 determines whether the received first adjusted image includes an image of the target of interest (S21). The request unit 313 performs this determination process, for example, as follows.

[0148] The request unit 313 first analyzes the received input language using a sentence analysis process to estimate the language indicating the target of attention. The request unit 313 estimates the language indicating the target of attention, for example, by extracting nouns included in the input language. The request unit 313 may input an instruction to the first language model 321 to extract the language indicating the target of attention from the input language, and the resulting response language may be the estimated result of the language indicating the target of attention.

[0149] The request unit 313 then determines whether an image of the target of interest indicated by the estimated language is included in the first adjusted image. The process of determining whether an image of a specific target expressed in language is included in an image can use, for example, image recognition technology such as CLIP (Contrastive Language-Image Pre-training) or YOLO (You Only Look Once). Alternatively, the request unit 313 may determine whether an image of the target of interest indicated by the estimated language is included in the first adjusted image based on a response result to a query input to the first language model 321 asking whether the image of the target of interest indicated by the estimated language is included in the first adjusted image.

[0150] If the request unit 313 determines that the image of the target indicated by the estimated language is not included in the first adjustment image (NO in S21), it requests additional data from the transmission data adjustment unit 212B (S22). In other words, if the target indicated by the received input language is not included in the first adjustment image generated by the first adjustment unit 222, the request unit 313 requests additional data from the transmission device 2B. At this time, the request unit 313 discards the received MM instruction data and notifies the second adjustment unit 312 of an instruction to wait for the next MM instruction data. In addition, the request unit 313 notifies the second adjustment unit 312 of an instruction to stop generating the second adjustment image based on the input language and the first adjustment image included in the MM instruction data.

[0151] 7, when the request receiving unit 224 receives a request for additional data from the request unit 313, the transmission data adjusting unit 212B notifies the setting unit 221 of an instruction to reset the important region. In response to the request from the request unit 313, the setting unit 221 resets the important region for the input image (S23). The setting unit 221 resets the important region for the input image as a new important region obtained by enlarging the important region set in S2.

[0152] Next, the first adjustment unit 222 adjusts the input image based on the reset important area to generate a new first adjusted image (S3).Then, the transmission control unit 223 transmits MM instruction data including the new first adjusted image to the second adjustment unit 312 and the request unit 313, and the processing of S11 is performed.

[0153] However, when the first adjustment unit 222 generates a new first adjustment image, the transmission control unit 223 may transmit the MM instruction data only to the second adjustment unit 312. That is, in this case, processing by the request unit 313 may not be performed. Furthermore, the transmission control unit 223 may transmit, as additional data, MM instruction data including a new combination consisting of the input language and the new first adjustment image to the second control unit 31B, instead of MM instruction data including only the new first adjustment image.

[0154] On the other hand, consider a case where the request unit 313 determines that the image of the target object indicated by the estimated language is included in the first adjustment image (YES in S21). In this case, the request unit 313 does not request additional data from the transmission data adjustment unit 212B, but instead notifies it to continue generating the second adjustment image based on the input language and the first adjustment image included in the received MM instruction data.

[0155] If the answer is YES in S21, or if a new first adjustment image is received, the response generation unit 311 uses the first language model 321 to generate a response language for the combination of the received input language and the second adjustment image (S12).

[0156] For example, in response to an input phrase such as "What is the person on the right doing?", the setting unit 221 sets an area corresponding to the right quarter of the input image as an important area based on the phrase "right," and the first adjustment unit 222 generates a first adjusted image based on the important area. The transmission control unit 223 transmits MM instruction data including the input phrase and the first adjusted image to the second adjustment unit 312 and the request unit 313.

[0157] The request unit 313 determines whether the first adjusted image contains an image of a person based on the word "person" included in the input language. If the request unit 313 determines that the first adjusted image contains an image of a person, it notifies the second adjustment unit 312 to continue the process of generating the second adjusted image. After generating the second adjusted image, the second adjustment unit 312 transmits the second adjusted image together with the input language to the response generation unit 311. As a result, the response generation unit 311 generates a response language based on the input language and the second adjusted image.

[0158] On the other hand, if the request unit 313 determines that the first adjustment image does not contain an image of a person, it requests additional data from the transmission data adjustment unit 212B. The setting unit 221 resets the right half of the input image, which is larger than the right quarter of the image that was initially set, as the important area. The first adjustment unit 222 then generates a new first adjustment image based on the reset important area (right half), and the transmission control unit 223 transmits MM instruction data including at least the new first adjustment image to the second control unit 31B.

[0159] The response system 1B can reduce the possibility of generating a response language indicating that the target of interest is not included in the input image, thereby reducing the possibility of the user being stressed by being presented with such a response language.

[0160] [Fourth embodiment] Fig. 9 is a block diagram showing an example of the configuration of a response system 1C of a fourth embodiment. Fig. 10 is a block diagram showing an example of a transmission data adjustment unit 212 and a local response generation unit 214 of the fourth embodiment.

[0161] As shown in FIG. 9 , the response system 1C differs from the response system 1 of the first embodiment in that it includes a transmission device 2C instead of the transmission device 2 and a reception device 3C instead of the reception device 3. The transmission device 2C differs from the transmission device 2 of the first embodiment in that it includes a first control unit 21C including a local response generation unit 214 instead of the first control unit 21. As shown in FIG. 10 , the local response generation unit 214 of this embodiment includes a caption generation unit 225. Also, as shown in FIG. 9 , the first storage unit 25 of the transmission device 2C stores a second language model 251. The reception device 3C differs from the reception device 3 of the first embodiment in that it includes a second control unit 31C including a final input language generation unit 314 instead of the second control unit 31.

[0162] <Processing Flow> An example of the processing flow in the response system 1C will be described using Figures 9 to 11. Figure 11 is a sequence diagram showing an example of the processing flow in the response system 1C. In Figure 11, the same processing as in Figure 3 will not be described.

[0163] In this embodiment, the transmission device 2C generates additional information for further narrowing down the important region using a second language model 251 available to the transmission device 2C. Specifically, as shown in Fig. 11 , after the acquisition unit 211 acquires the input language and the input image (S1), the local response generation unit 214 generates a response language for the combination of the input language and the input image using the second language model 251 stored in the first storage unit 25. In this embodiment, the caption generation unit 225 generates a caption for the input image as additional information using the second language model 251 (S31).

[0164] The second language model 251 is, for example, MM-LLM. However, the second language model 251 is selected so that the computational resources (required resources) required for its execution are equal to or less than the computational resources required for the execution of the first language model 321. As described above, this is because the computational resources of the first control unit 21C are scarcer than the computational resources of the second control unit 31C.

[0165] The input language for generating a caption is different from the input language transmitted to the receiving device 3C and is an input language (hereinafter referred to as the caption language) that includes content requesting caption generation. Examples of caption languages ​​include "What does this image represent?", "Generate a caption for this image," and "What is being photographed?" The caption language is preset and may be stored in the local response generation unit 214, or may be acquired separately from the input language via the acquisition unit 211. The caption generation unit 225 generates a caption for the input image as a response language for the combination of the caption language and the input image. The caption generation unit 225 then transmits the generated caption to the transmission control unit 223.

[0166] Next, the setting unit 221 sets an important region (S2), and the first adjustment unit 222 generates a first adjustment image (S3). The transmission control unit 223 then transmits MM instruction data including the input language, the first adjustment image, and a caption to the second control unit 31C (S32). Note that the process of S31 and the processes of S2 and S3 may be performed in parallel or in reverse order.

[0167] The final input language generation unit 314 generates a final input language based on the received input language and caption (S33). In this embodiment, the final input language generation unit 314 functions as a concretization language generation unit that generates, as the final input language, a concretization language that is more specific than the input language based on the received input language and caption. The final input language generation unit 314 transmits the generated concretization language as the final input language to the response generation unit 311. The response generation unit 311 uses the first language model 321 to generate a response language for the combination of the concretization language and the first adjustment image (S34). The combination of the concretization language and the first adjustment image is an example of data corresponding to the combination of the input language and the first adjustment image.

[0168] For example, consider a case where an image of a scene (scenery) with small fruits on a tree in the distance is used as an input image, the input language "What is that?" is input to the input unit 22, and the caption language described above is used. In this case, the caption generation unit 225 uses the second language model 251 to generate a caption for the input image, such as "There is a tree in the distance." The final input language generation unit 314 generates an instantiation language that includes an instruction to generate a response language in response to the input language "What is that?" and an instruction to create the response language taking into consideration the caption language "There is a tree in the distance" when generating the response language.

[0169] For example, "There are small red fruits on the tree, and they appear to be ripe" can be obtained as an analysis result of the first language model 321 based on the input language and the first adjusted image in which an important region including an image of a tree is set. In this embodiment, the response generation unit 311 can obtain an analysis result of the first language model 321 that also includes the caption language. Therefore, the response generation unit 311 can finally generate a response result that combines these two analysis results, such as "There is a tree in the distance, and it appears to have small red ripe fruits on it."

[0170] In this way, according to the response system 1C, a caption that takes into consideration the shooting scene included in the entire input image (the overall situation of the input image) is provided to the response generation unit 311. Therefore, the response generation unit 311 can generate a high-quality response language that explains the overall situation of the input image while providing a more detailed explanation of the target object ("tree" in the above example).

[0171] <Modification> The caption generated by the caption generation unit 225 may be transmitted to the setting unit 221. In this case, the setting unit 221 sets an area including an image of a tree as an important area based on the input language and the caption. This allows the setting unit 221 to set the important area accurately (small). This makes it possible to generate a high-quality response language as described above while reducing the amount of data in the first adjustment image.

[0172] The response generation unit 311 generates a response language based on the first adjusted image generated based on the important area set in this way and the concretization language. However, the response generation unit 311 may generate the response language using the first adjusted image and the input language instead of the concretization language. In this case, the second control unit 31C may not be provided with the final input language generation unit 314.

[0173] [Embodiment 5] Fig. 12 is a block diagram showing an example of the configuration of a response system 1D of embodiment 5. Fig. 12 is also a block diagram showing an example of a transmission data adjustment unit 212 and a local response generation unit 214D included in a transmission device 2D of embodiment 5. Fig. 12 omits the illustration of the input unit 22, the photographing unit 24, the acquisition unit 211, and the network 4. Fig. 12 also omits the illustration of the configuration of the reception device 3C shown in Fig. 9.

[0174] 12 , the response system 1D differs from the response system 1C of the fourth embodiment in that it includes a transmission device 2D instead of the transmission device 2C. The transmission device 2D differs from the transmission device 2C of the fourth embodiment in that it includes a first control unit 21D including a local response generation unit 214D. The local response generation unit 214D of the present embodiment includes a primary response generation unit 226.

[0175] <Processing Flow> An example of the processing flow in the response system 1D will be described with reference to Fig. 12 and Fig. 13. Fig. 13 is a sequence diagram showing an example of the processing flow in the response system 1D. In Fig. 13, the same processing as in Fig. 3 will not be described.

[0176] 13 , after the acquisition unit 211 acquires an input language and an input image (S1), the primary response generation unit 226 generates a primary response language (primary response) for the combination of the input language and the input image using the second language model 251 (S41). The primary response generation unit 226 transmits data indicating the generated primary response language and a presentation instruction to the presentation control unit 213. The presentation control unit 213 presents the received primary response language via the presentation unit 23 (S42). In addition, the primary response generation unit 226 transmits data indicating the generated primary response language to the transmission control unit 223.

[0177] Next, the setting unit 221 sets an important region (S2), and the first adjustment unit 222 generates a first adjustment image (S3). Then, the transmission control unit 223 transmits MM instruction data including the input language, the first adjustment image, and the primary response language to the second control unit 31C of the receiving device 3C (S43). Note that the processes of S41 and S42 and the processes of S2 and S3 may be performed in parallel or in reverse order.

[0178] The final input language generation unit 314 of the second control unit 31C generates a final input language based on the received input language and primary response language (S44). In this embodiment, the final input language generation unit 314 generates the final input language so that the response generation unit 311 can generate a response language taking the primary response language into consideration. The final input language generation unit 314 generates a final input language such as, for example, "Please explain the image in a format that is added to the following primary response. {Primary response} Answer:." The content of the primary response language is inserted in text format into "{Primary response}." The combination of the final input language including the primary response language and the first adjustment image is an example of data corresponding to a combination of an input language and a first adjustment image.

[0179] As described above, according to the response system 1D, the presentation control unit 213 presents the primary response language via the presentation unit 23 in response to the input language and input image before the response generation unit 311 generates a response language. Therefore, the user can understand the primary response language generated by the second language model 251 without waiting for the response language to be generated. Although the primary response language may be less accurate than the response language, the user can obtain a response to the inquiry at an earlier timing. Therefore, it is possible to reduce the possibility that the user will be stressed by waiting for the response language to be generated. Presenting the primary response language prior to the response language is particularly effective in cases where there is a delay in the presentation of the response language due to communication via the network 4.

[0180] Furthermore, the response generation unit 311 uses the first language model 321 to generate a response language for a combination of a final input language including the primary response language and a first adjustment image adjusted based on the important region, and the presentation control unit 213 presents the response language after presenting the primary response language. Therefore, the presentation control unit 213 can present to the user a response language with higher accuracy than the primary response language. Furthermore, the presentation control unit 213 can present to the user a concise and natural response language that does not overlap with the primary response language.

[0181] [Embodiment 6] Fig. 14 is a block diagram showing an example of the configuration of a response system 1E of embodiment 6. Fig. 14 is also a block diagram showing an example of a transmission data adjustment unit 212 and a local response generation unit 214E included in a transmission device 2E of embodiment 6. Fig. 14 omits the illustration of the input unit 22, the photographing unit 24, the acquisition unit 211, and the network 4. Fig. 14 also omits the illustration of the configuration of the reception device 3 shown in Fig. 1.

[0182] 14 , the response system 1E differs from the response system 1D of the fifth embodiment in that it includes a transmission device 2E instead of the transmission device 2D. The transmission device 2E differs from the transmission device 2D of the fifth embodiment in that it includes a first control unit 21E including a local response generation unit 214E. The local response generation unit 214E of the present embodiment includes a determination unit 227 in addition to a primary response generation unit 226. Furthermore, the response system 1E differs from the response system 1D in that it includes a receiving device 3.

[0183] <Processing Flow> An example of the processing flow in the response system 1E will be described with reference to Fig. 14 and Fig. 15. Fig. 15 is a sequence diagram showing an example of the processing flow in the response system 1E. In Fig. 15, the same processing as in Fig. 3 will not be described.

[0184] 15 , after the acquisition unit 211 acquires the input language and the input image (S1), the primary response generation unit 226 generates a primary response language (primary response) for the combination of the input language and the input image using the second language model 251 (S51). The primary response generation unit 226 transmits data indicating the generated primary response language to the determination unit 227.

[0185] In this embodiment, the primary response generation unit 226 uses the second language model 251 to generate, as a primary response language, content indicating whether the input image contains information necessary for the response generation unit 311 to generate a response language. In other words, the primary response generation unit 226 generates, as a primary response language, content indicating whether the target of interest indicated by the input language is contained in the input image. Therefore, the language used by the primary response generation unit 226 includes content that allows the second language model 251 to output the content. For example, the language includes content inquiring whether the input image contains information necessary for generating the response language. The language used by the primary response generation unit 226 is generated by the first control unit 21E (e.g., the primary response generation unit 226) analyzing the input language acquired by the acquisition unit 211.

[0186] For example, for an input language such as "What kind of fruit is this?", the language used by the primary response generation unit 226 can be configured as "Please respond as to whether or not the object for answering the following question is included in the input image. Question: What kind of fruit is this? Answer:" This allows the primary response generation unit 226 to generate, as a primary response language, content indicating whether or not the information required for generating a response language is included in the input image.

[0187] For example, when the acquisition unit 211 acquires an input language such as "What kind of fruit is this?" and an input image that does not include any fruit, the primary response generation unit 226 generates a primary response language stating that "The input image does not include the necessary information." On the other hand, when the acquisition unit 211 acquires the input language and an input image that includes fruit, the primary response generation unit 226 generates a primary response language stating that "The input image includes the necessary information."

[0188] Next, the determination unit 227 determines whether the target of interest indicated by the input language is included in the input image (S52). If the determination unit 227 determines that the target of interest is not included in the input image (NO in S52), it notifies the presentation control unit 213 of this fact. Upon receiving the notification from the determination unit 227, the presentation control unit 213 presents information indicating that the target of interest ("fruit" in the above example) is not included in the input image via the presentation unit 23 (S53).

[0189] On the other hand, if the determination unit 227 determines that the target is included in the input image (YES in S52), it notifies the setting unit 221. The setting unit 221 sets an important region upon receiving the notification from the determination unit 227 (S2). Thereafter, the processes of S3 to S7 are performed.

[0190] In this way, according to the response system 1E, if it is determined that a response language cannot be generated by the first language model 321 for the input language and input image, the user can be notified of this fact. This allows the user to know at an early stage that the expected response language will not be generated by the response generation unit 311. This reduces the possibility that the user will be stressed by having to wait for the generation of a response language when the expected response language cannot be generated.

[0191] When photographing with the transmitting device 2 (photographing unit 24) held in one's hand, there is a high possibility that the user is photographing while checking the photographing conditions via the photographing unit 24. Therefore, there is a high possibility that the target of interest is included in the captured input image. On the other hand, when photographing with the transmitting device 2, for example, hanging from the neck or placed in a breast pocket, there is a high possibility that the user is photographing without checking the photographing conditions via the photographing unit 24. Therefore, there is a high possibility that the target of interest is not included in the captured input image. Even in the latter case, by being able to present the fact that the target of interest is not included in the input image as described above, it is possible to prompt the user to retake the input image.

[0192] Furthermore, if it is determined that the first language model 321 cannot generate a response language, the MM instruction data is not transmitted to the response generation unit 311. This reduces the amount of data transmitted that is unnecessary for generating a response language.

[0193] [Modification] The response system described above includes a transmitting device and a receiving device that are communicatively connected via a network. However, the response system may be realized by a single device.

[0194] For example, a single device may include the acquisition unit 211, the setting unit 221, the first adjustment unit 222, and the response generation unit 311. Furthermore, for example, the single device may further include at least one of the transmission control unit 223, the request receiving unit 224, the presentation control unit 213, the local response generation units 214, 214D, and 214E, the second adjustment unit 312, the request unit 313, and the final input language generation unit 314.

[0195] Furthermore, the receiving device described above does not have to be realized by a cloud server, and may be realized by, for example, a conversational robot.

[0196] [Example of implementation by software] The functions of the response systems 1, 1A to 1E (hereinafter referred to as "devices") can be realized by a program that causes a computer to function as the device, and a program that causes a computer to function as each control block of the device (particularly each part included in the first control units 21, 21B, 21C, 21D, 21E, and each part included in the second control units 31, 31A, 31B, 31C).

[0197] In this case, the device includes a computer having at least one control device (e.g., a processor) and at least one storage device (e.g., a memory) as hardware for executing the program. The functions described in each of the above embodiments are realized by executing the program using the control device and storage device.

[0198] The program may be non-transitory and may be recorded on one or more computer-readable recording media. The recording media may or may not be included in the device. In the latter case, the program may be supplied to the device via any wired or wireless transmission medium.

[0199] In addition, some or all of the functions of each of the control blocks can be realized by logic circuits. For example, integrated circuits in which logic circuits that function as each of the control blocks are formed are also included in the scope of the present disclosure. In addition, the functions of each of the control blocks can also be realized by, for example, a quantum computer.

[0200] Furthermore, each process described in each of the above embodiments may be executed by AI (Artificial Intelligence). In this case, the AI ​​may run on the control device or on another device (for example, an edge computer or a cloud server).

[0201] [Summary] The response system according to aspect 1 of the present disclosure includes an acquisition unit that acquires an input language and an input image from a user, a setting unit that sets an important area in the input image that corresponds to the input language, a first adjustment unit that generates a first adjusted image by adjusting the input image based on the important area, and a response generation unit that uses a first language model to generate a response to a combination of the input language and the first adjusted image or data corresponding to the combination.

[0202] A response system according to a second aspect of the present disclosure is the same as in the first aspect, and further includes a presentation control unit that presents the important area set by the setting unit via a presentation unit.

[0203] A response system according to aspect 3 of the present disclosure is, in aspect 2, provided with a transmission control unit that transmits instruction data including the input language and the first adjustment image to the response generation unit, and the transmission control unit transmits the instruction data to the response generation unit when permission is obtained from the user to transmit information indicating the presented important area to the response generation unit.

[0204] In the response system according to aspect 4 of the present disclosure, in aspect 3, when the setting unit receives a user instruction to modify the important area set by the setting unit, the setting unit resets the important area.

[0205] The response system according to aspect 5 of the present disclosure is the same as in aspect 4, and repeats obtaining the user's correction instructions and presenting the reconfigured important area until the user's permission is obtained.

[0206] A response system according to aspect 6 of the present disclosure is any of aspects 1 to 5, wherein the response system comprises a transmitting device having the acquisition unit, the setting unit, and the first adjustment unit, and a receiving device having the response generation unit, the transmitting device and the receiving device being communicatively connected via a network, and the setting unit adjusts the setting range of the important area based on the performance or usage status of the first language model or the communication status with the receiving device.

[0207] A response system according to aspect 7 of the present disclosure is any one of aspects 1 to 6, wherein the setting unit sets the important area by using a demonstrative pronoun included in the input language.

[0208] A response system according to aspect 8 of the present disclosure is any one of aspects 1 to 7, wherein the setting unit sets the important area using an image that is included in the input image and that indicates a target of interest.

[0209] In a response system according to aspect 9 of the present disclosure, in any one of aspects 1 to 8, the setting unit sets the important area using language indicating the position of an object of interest contained in the input language.

[0210] In a response system according to aspect 10 of the present disclosure, in any one of aspects 1 to 9, the setting unit sets the important area using language included in the input language for identifying the object of attention.

[0211] A response system according to aspect 11 of the present disclosure is any of aspects 1 to 10, wherein the response system comprises a transmitting device having the acquisition unit, the setting unit, and the first adjustment unit, and a receiving device having the response generation unit, the transmitting device and the receiving device being communicatively connected via a network, the receiving device comprising a second adjustment unit that generates a second adjustment image by further adjusting the first adjustment image based on the input language, and the response generation unit generates a response to the combination of the input language and the second adjustment image.

[0212] A response system according to aspect 12 of the present disclosure is any of aspects 1 to 11, wherein the response system comprises a transmitting device having the acquisition unit, the setting unit, and the first adjustment unit, and a receiving device having the response generation unit, the transmitting device and the receiving device being communicatively connected via a network, and the receiving device having a request unit that requests additional data from the transmitting device when the target of interest indicated by the input language is not included in the first adjustment image.

[0213] In a response system according to aspect 13 of the present disclosure, in aspect 12, the setting unit resets the important area in response to a request from the request unit, and the first adjustment unit generates a new first adjusted image by adjusting the input image based on the reset important area.

[0214] A response system according to aspect 14 of the present disclosure is any of aspects 1 to 13, wherein the response system comprises a transmitting device having the acquisition unit, the setting unit, and the first adjustment unit, and a receiving device having the response generation unit, the transmitting device and the receiving device being communicatively connected via a network, and the transmitting device generates additional information for further narrowing down the important area using a second language model available to the transmitting device.

[0215] A response system according to aspect 15 of the present disclosure is in aspect 14, wherein the transmitting device includes a caption generation unit that uses the second language model to generate a caption for the input image as the additional information.

[0216] A response system according to aspect 16 of the present disclosure is, in aspect 15, wherein the transmitting device transmits the input language, the first adjusted image, and the caption to the receiving device, and the receiving device includes a concretization language generation unit that generates concretization language that is more specific than the input language based on the input language and the caption, and the response generation unit generates a response to the concretization language and the first adjusted image.

[0217] A response system according to aspect 17 of the present disclosure is any of aspects 1 to 16, wherein the response system comprises a transmitting device having the acquisition unit, the setting unit, and the first adjustment unit, and a receiving device having the response generation unit, the transmitting device and the receiving device being communicatively connected via a network, and the transmitting device comprises a primary response generation unit that generates a primary response to a combination of the input language and the input image using a second language model available to the transmitting device, and a presentation control unit that presents the primary response via a presentation unit.

[0218] A response system according to aspect 18 of the present disclosure, in any of aspects 1 to 17, includes a determination unit that determines whether the target of interest indicated by the input language is included in the input image, and a presentation control unit that, if the target of interest is not included in the input image, presents this fact via a presentation unit.

[0219] A response system according to aspect 19 of the present disclosure is any one of aspects 1 to 18, wherein the setting unit sets the important area based on the shooting condition of the input image in addition to the input language.

[0220] In the response system according to aspect 20 of the present disclosure, in any one of aspects 1 to 19, the acquisition unit acquires a video having a plurality of frames as the input image, and the setting unit sets the important area by regarding the frame corresponding to the time when the input language is input as the input image.

[0221] In the response system according to aspect 21 of the present disclosure, in any one of aspects 1 to 20, the acquisition unit acquires related language related to the target of interest indicated by the input language prior to the input language, and the setting unit sets the important area using the related language.

[0222] A response system according to aspect 22 of the present disclosure includes an acquisition step of acquiring an input language and an input image from a user, a setting step of setting an important area in the input image corresponding to the input language, a first adjustment step of generating a first adjusted image by adjusting the input image based on the important area, and a response generation step of generating a response to a combination of the input language and the first adjusted image or data corresponding to the combination using a first language model.

[0223] In addition, the response system according to each aspect of the present disclosure may be realized by a computer. In this case, the control program that realizes the response system on a computer by causing the computer to operate as each part (software element) of the response system, and the computer-readable recording medium on which the control program is recorded, also fall within the scope of the present disclosure.

[0224] [Additional Notes] The present disclosure is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the present disclosure. Furthermore, new technical features can be formed by combining the technical means disclosed in each embodiment.

[0225] [Cross-reference to related applications] This application claims the benefit of priority to Japanese Patent Application No. 2024-072729, filed on April 26, 2024, the entire contents of which are incorporated herein by reference.

[0226] 1, 1A to 1E Response system 2, 2B, 2C, 2D, 2E Transmission device 3, 3A, 3B, 3C Reception device 4 Network 23 Presentation unit 211 Acquisition unit 213 Presentation control unit 221 Setting unit 222 First adjustment unit 223 Transmission control unit 225 Caption generation unit 226 Primary response generation unit 227 Determination unit 311 Response generation unit 312 Second adjustment unit 313 Request unit 314 Final input language generation unit (concretized language generation unit)

Claims

1. A response system comprising: an acquisition unit that acquires an input language and an input image from a user; a setting unit that sets an important area in the input image that corresponds to the input language; a first adjustment unit that generates a first adjusted image by adjusting the input image based on the important area; and a response generation unit that uses a first language model to generate a response to a combination of the input language and the first adjusted image or data corresponding to the combination.

2. The response system according to claim 1, further comprising a presentation control unit that presents the important area set by the setting unit via a presentation unit.

3. A response system as described in claim 2, further comprising a transmission control unit that transmits instruction data including the input language and the first adjustment image to the response generation unit, wherein the transmission control unit transmits the instruction data to the response generation unit when permission is obtained from the user to transmit information indicating the presented important area to the response generation unit.

4. The response system according to claim 3, wherein the setting unit resets the important area when the setting unit receives an instruction from the user to modify the important area set by the setting unit.

5. The response system according to claim 4, wherein the process of obtaining the user's correction instruction and presenting the reconfigured important area is repeated until permission is obtained from the user.

6. The response system of claim 1, comprising a transmitting device having the acquisition unit, the setting unit and the first adjustment unit, and a receiving device having the response generation unit, wherein the transmitting device and the receiving device are communicatively connected via a network, and the setting unit adjusts the setting range of the important area based on the performance or usage status of the first language model or the communication status with the receiving device.

7. The response system according to claim 1, wherein the setting unit sets the important area by using demonstrative pronouns included in the input language.

8. The response system according to claim 1, wherein the setting unit sets the important area by utilizing an image that is included in the input image and that indicates a target of interest.

9. The response system according to claim 1, wherein the setting unit sets the important area using language that indicates the position of an object of interest contained in the input language.

10. The response system according to claim 1, wherein the setting unit sets the important area by using language included in the input language for identifying an object of interest.

11. The response system of claim 1, comprising a transmitting device having the acquisition unit, the setting unit and the first adjustment unit, and a receiving device having the response generation unit, wherein the transmitting device and the receiving device are communicatively connected via a network, and the receiving device comprises a second adjustment unit that generates a second adjusted image by further adjusting the first adjusted image based on the input language, and the response generation unit generates a response to the combination of the input language and the second adjusted image.

12. The response system of claim 1, comprising a transmitting device having the acquisition unit, the setting unit and the first adjustment unit, and a receiving device having the response generation unit, wherein the transmitting device and the receiving device are communicatively connected via a network, and the receiving device comprises a request unit that requests additional data from the transmitting device when the target of interest indicated by the input language is not included in the first adjustment image.

13. The response system described in claim 12, wherein the setting unit resets the important area in response to a request from the request unit, and the first adjustment unit generates a new first adjusted image by adjusting the input image based on the reset important area.

14. The response system of claim 1, comprising a transmitting device having the acquisition unit, the setting unit, and the first adjustment unit, and a receiving device having the response generation unit, wherein the transmitting device and the receiving device are communicatively connected via a network, and the transmitting device generates additional information for further narrowing down the important area using a second language model available to the transmitting device.

15. The response system according to claim 14, wherein the transmitting device comprises a caption generating unit that generates a caption for the input image as the additional information using the second language model.

16. The response system described in claim 15, wherein the transmitting device transmits the input language, the first adjusted image, and the caption to the receiving device, the receiving device includes a concretization language generation unit that generates concretization language that is more specific than the input language based on the input language and the caption, and the response generation unit generates a response to the concretization language and the first adjusted image.

17. The response system of claim 1, comprising: a transmitting device having the acquisition unit, the setting unit, and the first adjustment unit; and a receiving device having the response generation unit, wherein the transmitting device and the receiving device are communicatively connected via a network; and the transmitting device comprises: a primary response generation unit that generates a primary response to the combination of the input language and the input image using a second language model available to the transmitting device; and a presentation control unit that presents the primary response via a presentation unit.

18. A response system as described in claim 1, comprising a determination unit that determines whether the target of interest indicated by the input language is included in the input image, and a presentation control unit that, if the target of interest is not included in the input image, presents this fact via a presentation unit.

19. The response system according to claim 1, wherein the setting unit sets the important area based on the shooting condition of the input image in addition to the input language.

20. The response system of claim 1, wherein the acquisition unit acquires a video having a plurality of frames as the input image, and the setting unit sets the important area by regarding the frame corresponding to the time when the input language is input as the input image.

21. The response system of claim 1, wherein the acquisition unit acquires related words related to the target of interest indicated by the input language prior to the input language, and the setting unit sets the important area using the related words.

22. A response generation method including: an acquisition step of acquiring an input language and an input image from a user; a setting step of setting an important area in the input image corresponding to the input language; a first adjustment step of generating a first adjusted image by adjusting the input image based on the important area; and a response generation step of generating a response to a combination of the input language and the first adjusted image or data corresponding to the combination using a first language model.

Images