Electronic devices and information processing devices, and their control methods and programs

Abstract

This invention provides an electronic device, a control method, and a program for controlling its operation using arbitrary expressions in automated operations based on user instructions. [Solution] In the text generation AI service, the smartphone app receives a camera control voice instruction M111 from the user 101, interprets the received voice instruction, converts it into text data, and sends a camera setting item list request message M112 to the camera 102. The camera responds to the camera setting item list request message. When the smartphone app receives the camera setting item list request response M113 from the camera, it creates a camera setting item selection prompt M114 and sends it to the text generation AI service 104. The text generation AI service responds with a selection result M115 indicating which setting items should be changed in order to take a photo that meets the photographer's requirements.

Description

Technical Field

[0001] The present invention relates to an electronic device, an information processing apparatus, a control method thereof, and a program that perform automatic processing based on an instruction from a user.

Background Art

[0002] In recent years, a system that automatically starts processing in response to a user's voice input has reached the stage of practical use. This technology significantly reduces the need for manual operations and enables more intuitive and rapid control of electronic devices. Patent Document 1 discloses a shooting instruction method in which when a user utters a password (e.g., "Take a photo") to instruct the start of shooting, the voice is recognized by a voice processing unit and used as a shooting trigger.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the technology described in Patent Document 1 above, voice commands are limited to pre-registered phrases, and users had to memorize and use specific phrases.

[0005] The present invention has been made in view of such circumstances, and an object thereof is to provide an electronic device capable of controlling processing with an instruction of an arbitrary expression in automatic processing based on an instruction from a user.

Means for Solving the Problems

[0006] To solve the above problems, according to one aspect of the present invention, an electronic device that performs processing according to a set value comprises: a first transmitting means that transmits information of the arbitrary text information and the setting item of the electronic device to a selection means that selects at least one setting item of the electronic device based on arbitrary text information input as an instruction from a user; a first receiving means that receives the setting item from the selection means; a second transmitting means that transmits information of a configurable setting value for the setting item received by the first receiving means to a determination means that determines a setting value for performing processing according to an instruction from a user; a second receiving means that receives the setting value from the determination means; and a control means that controls the device to perform processing based on the setting value received by the second receiving means.

[0007] Furthermore, according to another aspect of the present invention, an electronic device that performs processing according to a set value comprises a third transmitting means for transmitting information of a setting item of the electronic device to an information processing device, and a third receiving means for receiving the set value from the information processing device, wherein the information processing device comprises a fourth receiving means for receiving information of a setting item of the electronic device, a first transmitting means for transmitting the arbitrary text information and information of the setting item of the electronic device to a selection means for selecting at least one setting item of the electronic device based on arbitrary text information input as an instruction from a user, a first receiving means for receiving the setting item from the selection means, a second transmitting means for transmitting information of a configurable setting value for the setting item received by the first receiving means to a determination means for determining a setting value for performing processing according to an instruction from a user, a second receiving means for receiving the setting value from the determination means, and a fourth transmitting means for transmitting the setting value received by the second receiving means to the electronic device, wherein the device performs processing based on the setting value received by the third receiving means. [Effects of the Invention]

[0008] According to the present invention, it is possible to provide an electronic device that can control processing using arbitrary expressions in automated processing based on user instructions. [Brief explanation of the drawing]

[0009] [Figure 1] This diagram illustrates the process of determining camera settings using a generation AI service in a camera according to the first embodiment. [Figure 2] This is a diagram showing the camera configuration according to the first embodiment. [Figure 3] This is an example of a smartphone application screen used in the first embodiment. [Figure 4] In the first embodiment, this is an example of a prompt for causing the AI ​​service to generate camera setting items to select. [Figure 5] This is a flowchart illustrating the process of interpreting a message sent from a generating AI service in the first embodiment. [Figure 6] In the first embodiment, this is an example of a prompt for causing the AI ​​service to generate camera settings to select them. [Figure 7] This diagram illustrates a flowchart for determining printer settings using a generation AI service in a printer according to the second embodiment. [Figure 8] This is an example of the printer driver PC application screen used in the second embodiment. [Figure 9] In the second embodiment, this is an example of a prompt for causing the AI ​​service to generate printer settings to select a printer setting item. [Figure 10] In the second embodiment, this is an example of a prompt for the AI ​​service to select printer settings. [Modes for carrying out the invention]

[0010] The present invention will be described in detail below with reference to the attached drawings, based on exemplary embodiments thereof. Note that the following embodiments do not limit the invention to the claims. Furthermore, while multiple features are described in the embodiments, not all of them are essential to the invention, and the multiple features may be combined arbitrarily. In addition, in the attached drawings, the same or similar configurations are given the same reference numeral, and redundant descriptions are omitted.

[0011] In the following embodiments, the present invention will be described in relation to cases where it is implemented using a camera with pan-tilt functionality (first embodiment) or a printer (second embodiment) as an example of an electronic device. However, the electronic device is not limited to these. For example, it may be other electronic devices such as home appliances like refrigerators and microwave ovens, or office equipment such as multifunction printers. Also, in the first embodiment, a camera with pan-tilt functionality was used as an example of an imaging device, but other imaging devices may be used. For example, digital cameras, video cameras, smartphones, tablets, wearable cameras, smartwatches, smart glasses, webcams, security cameras, game consoles, robots, drones, dashcams, etc. Examples of printers include inkjet printers, laser printers, 3D printers, sublimation printers, portable printers, and large-format printers.

[0012] <First Embodiment> In this embodiment, we will explain an example of a process for performing the desired shooting even when the user's (photographer's) instructions to the camera include colloquial expressions such as "Take a picture with A in the center," "Take pictures of the children," or "Take lots of pictures for about 5 minutes."

[0013] In this embodiment, as a method for performing fine-grained device control from arbitrary natural language expressions, the input natural language is interpreted by a generating AI service and converted into device control commands, thereby controlling the device. In this case, when sending instructions in arbitrary natural language to the generating AI service, the setting items provided by the device and the options for setting values ​​within those items are added to the prompt and provided to the generating AI service, allowing for detailed and appropriate device settings. However, some generating AI services are paid services, and especially when using APIs, some employ a pay-per-use system where usage fees are determined according to the number of characters or words in the text. Therefore, we will also consider ways to reduce costs by reducing the number of characters in the prompt text sent to the generating AI service. The generating AI service is composed of Large Language Models (LLMs). A Large Language Model is a deep learning model composed of an artificial neural network with a large number of parameters, which generates and outputs appropriate responses to instructions (prompts) in natural language.

[0014] (Controlling the camera using natural language instructions) Figure 1 illustrates the flow of controlling the camera 102 using natural language voice commands from user 101. In this embodiment, a smartphone application 103 is used as an information processing device that receives and interprets voice from user 101. If the camera 102 is equipped with voice reception, voice analysis, and text conversion functions, a configuration without using the smartphone application 103 is also possible. 104 is a text generation AI service, and is assumed to be an internet-based generation AI service such as ChatGPT.

[0015] M111 is a voice instruction for camera control by the voice of user 101. For example, an instruction like "Take a picture of the delicious-looking food". The voice instruction for camera control M111 is first received by the smartphone app 103, and the smartphone app 103 interprets the received voice and converts it into text data. There are various methods for converting voice into text. For an Android app, the Speech Recognizer library can be used. A web service for converting voice data into a string can also be used. For example, there is Speech-to-Text provided by Google (https: / / cloud.google.com / speech-to-text).

[0016] M112 is a camera setting item list request message for requesting a list of camera setting items from the smartphone app 103 to the camera 102. When the smartphone app 103 receives the camera control voice instruction M111 from the user 101, it sends the camera setting item list request message M112 to the camera 102.

[0017] M113 is a response to the camera setting item list request message M112 in the camera 102, which is a camera setting item list request response. The response content is a list of camera setting items determined by the text generation AI service 104 and the current setting values for each camera setting item. The reason for including the current setting values in the response content is to respond to requests from the user 101 to make relative changes from the current camera settings, such as "Take a picture a little brighter than the current one". Also, the camera setting items included in the camera setting item list request response M113 may be changed according to the state of the camera and the subject detection situation. For example, when the user 101 has set the shutter speed manually, this manual setting can be prioritized and the shutter speed can be removed from the camera setting item list.

[0018] M114 is a camera setting item selection prompt that causes the text generation AI service 104 to select at least one camera setting item from the smartphone app 103. When the smartphone app 103 receives the camera setting item list request response M113 from the camera 102, it creates the camera setting item selection prompt M114 and sends it to the text generation AI service 104. The camera setting item selection prompt M114 includes the camera control voice instruction M111 and the contents of the camera setting item list request response M113, as shown in Figure 4(A).

[0019] M115 is the result of the camera setting selection (first stage response) from the text generation AI service 104, which received the camera setting selection prompt M114. As shown in Figure 4(B), the text generation AI service 104 responds with the selection result for "Which setting should be changed in order to take a photo that meets the photographer's requirements?"

[0020] M116 is a camera setting list request message that the smartphone application 103 requests from the camera 102 after receiving the camera setting item selection result M115. The smartphone application 103 extracts the camera setting items that need to be changed from the contents of the camera setting item selection result M115. Then, it sends the camera setting list request message M116 to the camera 102, which is a request for a list of configurable camera setting values ​​for the extracted camera setting items.

[0021] M117 is a camera setting value list request response that camera 102 sends to smartphone app 103. In response to receiving the camera setting value list request message M116 from smartphone app 103, camera 102 creates a list of configurable settings for the target camera setting item and sends it to smartphone app 103.

[0022] M118 is a camera setting determination prompt that prompts the smartphone application 103, which has received the camera setting list request response M117, to cause the text generation AI service 104 to determine an appropriate setting from the setting list. This camera setting determination prompt M118 includes a camera control voice instruction M111 to ensure that the setting is as intended by the user 101.

[0023] M119 is the result of selecting camera settings (second stage response) from the text generation AI service 104, which received the camera setting determination prompt M118.

[0024] M120 is a camera setting change message that allows the smartphone app 103 to set the camera settings determined by the camera setting selection result M119 from the text generation AI service 104 to the camera 102. When the camera 102 receives this camera setting change message M120, it changes the camera settings and performs a shooting operation.

[0025] As described above, the response from the text generation AI service 104 in the first stage consists of several camera setting items, and the response from the text generation AI service 104 in the second stage consists of the setting values ​​for the setting items narrowed down in the first stage. This allows for obtaining a response with appropriate settings while keeping the amount of text (token amount) communicated with the text generation AI service 104 to a minimum.

[0026] (Camera configuration) Figure 2 is a block diagram illustrating the internal configuration of the camera 102 according to this embodiment.

[0027] The lens barrel 200 has an imaging optical system and an image sensor that acquires image data based on the light beam from the imaging optical system, and is attached to the fixed part (not shown) of the camera 102 via a rotating mechanism that can rotate it.

[0028] The lens unit 201 includes a zoom unit and a focus unit. The zoom unit includes a zoom lens that performs magnification. The focus unit includes a lens that adjusts focus. The lens unit 201 is driven and controlled by the lens drive unit 210.

[0029] The imaging unit 202 includes an image sensor, which receives light beams incident through each lens group and generates charge information corresponding to the amount of light as analog image data. This analog image data is output to the image processing unit 212.

[0030] The control box 204 includes a control microcontroller for controlling the imaging lens group contained in the lens barrel 200, the tilt rotation unit 205, and the pan rotation unit 206. In this embodiment, the control box 204 is located within the fixed part of the camera 102, and the control box 204 remains fixed even when the lens barrel 200 is driven to pan and tilt.

[0031] The lens barrel drive unit 211 rotates the lens barrel 200 in the tilt and pan directions by driving the tilt rotation unit 205 and the pan rotation unit 206. The lens barrel drive unit 211 is controlled by the control unit 217.

[0032] Furthermore, the camera 102 uses an aperture control unit, a sensor gain control unit, and a shutter control unit (not shown) to perform exposure control so that the subject is at an appropriate brightness.

[0033] The image processing unit 212 converts the analog image data input from the imaging unit 202 into digital image data using A / D conversion. The image processing unit 212 applies image processing such as distortion correction, white balance adjustment, and color interpolation to this digital image data and outputs the digital image data after processing. The digital image data output from the image processing unit 212 is converted into a recording format such as JPEG format by the recording unit 213 and transmitted to the RAM 219 and recording medium 214.

[0034] The recording unit 213 records the compressed image signal, compressed audio signal, and other control data related to shooting, generated by the image processing unit 212 and the audio processing unit 215, onto the recording medium 214. If the audio signal is not compressed, the control unit 217 transmits the audio signal generated by the audio processing unit 215 and the compressed image signal generated by the image processing unit 212 to the recording unit 213 for recording onto the recording medium 214.

[0035] The recording medium 214 is a recording medium built into the camera 102, but it may also be a removable recording medium. Various data such as compressed image signals, compressed audio signals, and audio signals generated by the camera 102 can be recorded on the recording medium 214. For this reason, a medium with a larger capacity than the ROM 220 is generally used for the recording medium 214. For example, the recording medium 214 may be any type of recording medium, such as a hard disk, optical disk, magneto-optical disk, CD-R, DVD-R, magnetic tape, non-volatile semiconductor memory, or flash memory.

[0036] The audio processing unit 215 performs audio-related processing, such as optimizing the input digital audio signal. The audio signal processed by the audio processing unit 215 is then transmitted to the RAM 219 by the control unit 217.

[0037] The display unit 216 has the function of outputting visually recognizable information, such as an LCD or LED, or outputting sound, such as from a speaker, and displays various types of information.

[0038] The control unit 217 consists of, for example, a CPU (MPU), memory (DRAM, SRAM), and non-volatile memory (EEPROM). The control unit 217 executes various processes (programs) to control each block of the camera 102 and to control data transfer between each block.

[0039] The communication unit 218 communicates with the camera 102 and an external device such as a smartphone application 103, sending and receiving data such as audio signals, image signals, compressed audio signals, compressed image signals, and text messages. Furthermore, when the camera 102 detects an abnormal condition, it sends information to the external device to notify it of the internal state of the camera, such as error information. The communication unit 218 includes, for example, wireless communication modules such as an infrared communication module, a Bluetooth communication module, a wireless LAN communication module, a WirelessUSB, and a GPS receiver.

[0040] RAM219 temporarily stores the image signal and audio signal obtained by the image processing unit 212 and the audio processing unit 215.

[0041] ROM220 is an electrically erasable and recordable memory that stores constants, programs, and other information for the operation of the control unit 217.

[0042] The operation unit 221 is an input device that accepts various operations from the user 101. For example, the operation unit 221 can be a touch panel or physical buttons. The touch panel, for example, is provided on the display surface of the display unit 216 and integrated with the display unit 216. The operation unit 221 and the display unit 216 may or may not be detachable from the camera 102. Furthermore, it may be implemented as an application for a general-purpose computing device such as a smartphone.

[0043] The audio input unit 222 acquires audio signals from the microphone provided on the camera 102, converts them from analog to digital, and transmits them to the audio processing unit 215.

[0044] The subject detection unit 223 detects subjects included in the captured image and determines the attributes of the subjects. The subject detection unit 223 detects the faces and bodies of the subjects. In the face detection process, a pattern for determining the faces of subjects is predetermined, and areas in the captured image that match this pattern can be detected as the faces of the subjects.

[0045] Furthermore, a confidence score indicating the likelihood that the subject is indeed a face is calculated simultaneously. This confidence score is calculated based on factors such as the size of the face region within the image and the degree of match with the face pattern. Similarly, for object recognition, objects that match pre-registered patterns can be recognized.

[0046] Another method involves extracting feature subjects using histograms of hue, saturation, etc., within the captured image. In this case, for the image of the subject captured within the field of view, the distribution derived from the histogram of hue, saturation, etc., is divided into multiple intervals, and the captured images are classified according to each interval. For example, histograms of multiple color components are created for the captured image, and the image is divided into bell-shaped distribution ranges. Images captured in regions belonging to the same combination of intervals are classified, and the image region of the subject is recognized. By calculating an evaluation value for each recognized image region of the subject, the image region of the subject with the highest evaluation value can be determined as the main subject region. Using the above method, information on each subject can be obtained from the captured information.

[0047] The subject detection unit 223 further performs attribute estimation of the detected subject. Attribute estimation is performed in the detected face region using a predetermined determination formula based on edge information such as eyes and mouth, and contours. In other embodiments, machine learning may be used, and the method and content are not specified. In this embodiment, the type of subject, i.e., human, cat, or other biological classification, is estimated. The attributes to be estimated may also include other attributes, such as race, face orientation, face shape, organ / hair color, and presence or absence of accessories (mask, glasses, sunglasses, eye patch, bandage, collar, etc.).

[0048] The image processing unit 212 and the audio processing unit 215 read the image signal and audio signal temporarily stored in the RAM 219, encode the image signal and the audio signal respectively, and generate a compressed image signal and a compressed audio signal.

[0049] (Example screen from smartphone app 103) Figure 3 shows an example screen of the smartphone application 103.

[0050] The display 301 of the smartphone on which the smartphone app 103 is installed will show the items 302 to 305 described later.

[0051] Item 302 is a voice input button. When this voice input button 302 is tapped, the system enters voice input acceptance mode. Tapping the voice input button 302 again ends the voice input acceptance mode. The voice input from the start to the end of the voice input acceptance mode is considered as a single shooting instruction.

[0052] Item 303 is a voice-to-text display of shooting instructions, which is a text conversion of the voice instructions entered into the smartphone app 103.

[0053] Item 304 is a preview display of an image taken by camera 102 based on a shooting instruction from user 101. The latest image is retrieved from camera 102 and displayed on the screen.

[0054] Item 305 is a response message displayed from the smartphone app 103 when a shooting instruction is received. The camera settings at the time of shooting may also be displayed in this response message display item 305. In this case, the user 101 can check the camera settings displayed in the response message display item 305, manually change the camera settings, and then take the picture.

[0055] (Communication between camera 102 and smartphone app 103) Next, we will explain the communication between the camera 102 and the smartphone application 103. The HTTP protocol is used as the communication method. The communication unit 218 of the camera 102 has the function of receiving and interpreting the HTTP protocol.

[0056] The following five steps are involved in the communication between camera 102 and smartphone app 103. (1) Camera setting item list request message M112 (2) The response to M112 is the camera setting item list request response M113 (3) Camera setting list request message M116 (4) The response to M116 is the camera setting value list request response M117 (5) Camera setting change message M120 First, here is an example of an HTTP request for the camera settings item list request message M112: GET http: / / [IP address]:[port number] / cameraapi / camerasettinglist HTTP / 1.1

[0057] When camera 102 receives this HTTP request, it lists the camera settings to include in the response message. The camera settings listed here do not have to be all of the settings available in camera 102, but only a portion of them. Camera 102's specifications may pre-define the camera settings to be included in the response. Alternatively, camera 102 may also pre-define the camera settings to be included in the response depending on the situation it is detecting. For example, if no face is detected during subject detection, settings related to faces (such as settings to prioritize AF on faces, settings to increase face exposure, etc.) may not be included.

[0058] Once the camera settings to be included in the response are determined, camera 102 creates a response message M113, which is the camera settings list request response message. In this embodiment, the camera settings to be included in the response are "Shutter Speed", "Aperture", "ISO", "Exposure Compensation", "White Balance", "Continuous Shooting Mode", "Contrast", "Color Filter Effect", "HDR Shooting", and "Strobe Flash". The response data format is JSON and will have the following content. { “Shutter speed”:{ "Current value": "1 / 60" }, “Aperture”:{ "Current value": "F4.0" }, “ISO”:{ "Current value": "ISO800" }, “Exposure compensation”:{ "Current value": "±0" }, “White balance”:{ "Current value": "White fluorescent lamp" }, "Continuous shooting mode": { "Current value": "1 photo taken" }, "contrast":{ "Current value": "0" }, “Color filter effect”:{ "Current value": "None" }, “HDR shooting”: "Current value": "OFF" }, “Strobe flash”:{ "Current value": "Do not do" } }

[0059] The following is an example of an HTTP request for the camera settings list request message M116.

[0060] [In the case of shutter speed] GET http: / / [IP address]:[port number] / cameraapi / camerasetting / shutterspeed HTTP / 1.1

[0061] [In the case of white balance] GET http: / / [IP address]:[port number] / cameraapi / camerasetting / wb HTTP / 1.1

[0062] In this embodiment, an HTTP request URL is provided for each camera setting item. The smartphone application 103 extracts the camera setting items to be changed based on the camera setting item selection result M115 received from the text generation AI service 104, and sends an HTTP request to the camera 102 using the URL corresponding to the extracted camera setting item.

[0063] When camera 102 receives this HTTP request, it lists the camera settings to be included in the response message. The camera settings listed may not be all of the settings present in camera 102, but only a portion of them, similar to when creating the M113 camera setting item list request response message. For example, regarding aperture, even if the camera can stop down beyond f32, stopping down beyond f32 may prevent proper exposure even when using the maximum shutter speed and ISO sensitivity. In this case, the camera settings included in the response may be limited to f32.

[0064] Once the camera settings to be included in the response have been determined, camera 102 creates a camera setting value list request response M117. In this embodiment, JSON format is used, and the content will be as follows.

[0065] [In the case of aperture] { "Current value": "F4.0", “Settings List”: [“F1.8”, “F2.0”, “F2.5”, “F2.8”, “F3.5”, “F4.0”, “F4.5”, “F5.6”, “F8.0”, “F11”, “F16”, “F22”, “F32”] }

[0066] The following is an example of an HTTP request for the camera setting change message M120. Instead of a GET command, a PUT command is issued to the URL used in the request for the camera setting list request message M116.

[0067] [In the case of shutter speed] PUT http: / / [IP address]:[port number] / cameraapi / camerasetting / shutterspeed HTTP / 1.1 In this embodiment, the data format for the camera settings set using the PUT command is JSON format and will be as follows: { “value”: “1 / 60” }

[0068] (Example of camera setting selection prompt M114) Figure 4(A) shows an example of the camera setting item selection prompt M114 mentioned above. First, the text generation AI service 104 is given the role of responding with appropriate camera settings based on the photographer's requests, like a professional photographer. The method of response is then limited. In this example, an upper limit is given on the number of items that can be selected from the setting items, but it is not always necessary to set an upper limit. In addition, the format of the response text is also specified in detail for the text generation AI service 104. In Figure 4(A), the format is limited to "Item 1, Item 2, Item 3...". This is because if the text generation AI service 104 does not limit the format in this way, it will return a response in natural language, making it difficult for the smartphone application 103 to programmatically interpret the response. Also, in the example in Figure 4(A), the camera setting item name is included as is in the prompt, but it can be replaced with other words. For example, if the setting item name is "Highlight side - Tone priority", it can be changed to an expression such as "Highlight clipping reduction". Alternatively, it can be replaced with an item that encompasses multiple camera setting items. For example, you could combine "color temperature" and "contrast" to create a phrase like "warmth in the photo." However, as mentioned above, if you change the wording of a camera setting, you will also need to change the wording of that camera setting.

[0069] Figure 4(B) shows an example of the camera setting selection result M115 sent from the text generation AI service 104 to the smartphone application 103. The camera setting items "aperture," "exposure compensation," and "contrast" are separated by half-width commas in the message, which is in the same format as specified in the prompt in Figure 4(A).

[0070] (Sequence for interpreting messages in smartphone app 103) Figure 5 shows the sequence in which the smartphone app 103 receives the camera setting selection result M115 and interprets the message.

[0071] In S501, the received camera setting selection result M115 string is split by a half-width comma and stored in a string array. In the example in Figure 4(B), "aperture," "exposure compensation," and "contrast" are placed in the respective elements of the string array.

[0072] String array[0] = "Aperture", String array[1] = "Exposure compensation", String array[2] = "Contrast" This is the result.

[0073] In S502, this process initializes the count variable i for a loop that checks whether each string packed into the string array is a string that exists in the camera settings.

[0074] S503 is the starting point of a loop that checks whether each string packed into the string array is a string that exists in the camera settings. The loop repeats for the number of elements in the string array. In this embodiment, the loop will repeat three times for i=0,1,2.

[0075] In S504, check if the string in the string array [i] is a valid camera setting item. The list of camera setting items is obtained from the camera in the camera setting item list request response M113. Check if there is an item in the camera setting item list that matches the string in the string array [i].

[0076] S505, if the string in the string array [i] in S504 is a camera setting item name, adds this string to the setting change item list. The setting change item list is maintained by the smartphone application 103 in the form of a variable, database, etc., and based on this information, a camera setting value list request message M116 is created and sent to the camera 102.

[0077] S506 is the process of incrementing the loop counter variable i by 1.

[0078] In S507, if the loop counter variable i has advanced by the number of elements in the string array, the loop is exited and the camera setting item selection result M115 interpretation process is terminated.

[0079] The camera setting selection result M115 is generated by the text generation AI service 104 and may not be in the expected format, the camera setting item may be invalid, or the answer may not be appropriate. If the number of elements in the setting change item list becomes 0, or if the number of elements in the string array and the number of elements in the setting change item list are different, the camera setting selection prompt M114 may be sent again to the text generation AI service 104.

[0080] (Example of camera setting confirmation prompt M118) Figure 6(A) shows an example of the camera setting value determination prompt M118 mentioned above. Similar to the camera setting item selection prompt M114, the text generation AI service 104 is given the role of answering with appropriate camera settings based on the photographer's request, and the format of the answer is also specified. Furthermore, Figure 6(A) includes a list of camera setting values ​​for the camera setting item selected in the camera setting item selection result M115.

[0081] Figure 6(B) shows an example of the camera setting selection result M119 sent from the text generation AI service 104 to the smartphone application 103. As specified in the prompt in Figure 6(A), it represents a combination of camera setting values ​​in the format "setting item: setting value", such as "aperture: F2.8", with each setting item separated by a half-width comma.

[0082] (Summary of the first embodiment) As described above, with the configuration of this embodiment, even if the user's instructions are vague and colloquial, the desired shooting operation can be performed. Furthermore, the query to the text generation AI service 104 is made in two stages: the camera setting item and the camera setting value for the selected camera setting item. This reduces the amount of text compared to querying all combinations of camera setting items and camera setting values ​​present in camera 102 at once.

[0083] Although the method described here uses speech recognition with the voice input unit 222 and the voice processing unit 215 to input instructions using arbitrary text information, other configurations may be used for inputting arbitrary text instructions. For example, strings may be directly input using a character input device such as a keyboard. Alternatively, the system may accept strings via a chat application running on another device such as a smartphone.

[0084] <Second Embodiment> The second embodiment uses a printer as an example.

[0085] (Controlling the printer using natural language instructions) Figure 7 illustrates the flow of how printer 702 is controlled by natural language instructions from printer user 701. 703 is a printer driver PC application equipped with keyboard text input functionality, as well as printer 702 settings and print command functions. 704 is a text generation AI service, assumed to be an internet-based service such as ChatGPT.

[0086] M711 is a printer control instruction from the printer user 701. Here, it is a natural language instruction entered via keyboard. For example, an instruction such as "Print New Year's cards."

[0087] M712 is a printer setting item determination prompt used by the printer driver PC application 703 to instruct the text generation AI service 704 to determine the printer setting items. The printer setting item determination prompt M712 includes the printer control instruction M711 from the printer user 701 and a list of printer setting items. Since the printer driver PC application 703 knows the settings items that can be set on the printer 702, it is possible to create a list of printer setting items without querying the printer 702 for the settings items.

[0088] M713 is the result of selecting printer settings from the text generation AI service 704, which received the printer setting selection prompt M712.

[0089] M714 is a printer setting determination prompt that allows the printer driver PC application 703 to instruct the text generation AI service 704 to determine the appropriate setting value from the list of printer setting values. This printer setting determination prompt M714 contains a list of printer setting values ​​for the printer setting item selected in the printer setting item selection result M713, and the contents of the printer control instruction M711. Since the printer driver PC application 703 has knowledge of the setting values ​​to be set on the printer 702, it is possible to create a list of printer setting values.

[0090] M715 is the result of selecting printer settings from the text generation AI service 704, which received the printer setting determination prompt M714.

[0091] M716 is a printer print command message issued by the printer driver PC application 703, which receives the printer setting selection result M715 from the text generation AI service 704, to set the printer settings on printer 702 and then issue a print command. When printer 702 receives this printer print command message M716, it changes the printer settings as instructed and performs the print operation.

[0092] (Example screen of printer driver PC application 703) Figure 8 shows an example screen of the printer driver PC application 703.

[0093] On the PC display 801, where the printer driver PC application 703 is open, the items 802-808 described below will be displayed.

[0094] Item 802 displays the filename of the print file. This filename can also be sent to the text generation AI service 704 to provide information about what kind of file is being printed.

[0095] Item 803 is the print preview screen.

[0096] Item 804 is a chat area for the printer driver PC application 703. The chat area includes an input section for the printer user 701, a display section for the input content, and a display section for the response from the printer driver PC application 703.

[0097] Item 805 displays the input content of printer user 701.

[0098] Item 806 is a message displayed from the printer driver PC application 703. This may display messages such as "Please tell me the number of pages" to request additional printer control instructions 711 from the printer user 701 if there are any shortcomings in the printer control instructions 711. Alternatively, it may display the selected printer settings and their values ​​to allow the printer user 701 to confirm that the printer settings are as intended.

[0099] Item 807 is a text input section for the printer user 701 to input printer control instructions 711.

[0100] Item 808 is the print button. When the printer user 701 clicks this print button, the printer driver PC application 703 sends a printer print command message M716 to the printer 702, and the printing operation begins.

[0101] (Example from the printer settings selection prompt M712) Figure 9(A) shows an example of the printer setting item selection prompt M712 mentioned above. First, the text generation AI service 704 is given the role of answering with appropriate printer settings based on the request of the printer user 701. The message giving this role includes the file name of the file to be printed, providing information for the text generation AI service 704 to determine the print settings. The response method is then limited with the phrase "The following is a bulleted list...". The format of the response text is specified in detail to the text generation AI service 704. In Figure 9(A), it is limited to the format "Item 1, Item 2, Item 3...". This is to facilitate the implementation of string interpretation of the printer setting item selection result M713 in the printer driver PC application 703, which receives the printer setting item selection result M713, the response from the text generation AI service 704.

[0102] Figure 9(B) shows an example of the printer setting item selection result M713 sent from the text generation AI service 704 to the printer driver PC application 703. The setting items for printer 702, "Paper Size," "Paper Type," and "Color Mode," are separated by half-width commas in the message, which is in the same response format as specified in Figure 9(A).

[0103] (Example of the printer setting confirmation prompt M714) Figure 10(A) shows an example of the printer setting value determination prompt M714 mentioned above. Similar to the printer setting item selection prompt M712, it is responsible for prompting the text generation AI service 704 to provide appropriate printer settings based on the printer user 701's request, and for providing the file name of the file to be printed. It also specifies the format of the response in detail. Furthermore, Figure 10(A) includes a list of printer setting values ​​in the printer setting item selection result M713.

[0104] Figure 10(B) shows an example of the printer setting selection result M714 sent from the text generation AI service 704 to the printer driver PC application 703. It represents a combination of printer setting values ​​for a printer setting item in the format "setting item: setting value", such as "paper size: postcard", with each setting item separated by a half-width comma.

[0105] (Summary of the second embodiment) As described above, with the configuration of this embodiment, even if the user's instructions are vague and colloquial, the desired printing operation can be performed. Furthermore, the query to the text generation AI service 704 is made in two stages: first, the printer setting item, and then, the printer setting value for the selected printer setting item. This reduces the amount of text compared to querying all combinations of printer setting items and printer setting values ​​present in the printer 702 at once.

[0106] <Other Embodiments> The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0107] <Summary> The above embodiments include the following configuration.

[0108] (Composition 1) An electronic device that performs processing according to a set value, A first transmission means transmits the arbitrary text information and information about the setting item of the electronic device to a selection means that selects at least one setting item of the electronic device based on arbitrary text information received as input as instructions from the user, A first receiving means that receives the setting item from the selection means, A second transmission means transmits information about the configurable setting values ​​for the setting item received by the first receiving means to a determination means that determines the setting value for executing a process in response to instructions from the user, A second receiving means for receiving the set value from the determination means, An electronic device characterized by having a control means for controlling the device to perform processing based on the set value received by the second receiving means.

[0109] (Configuration 2) An electronic device that performs processing according to a set value, A third transmission means for transmitting information of the setting items of the electronic device to an information processing device, It has a third receiving means for receiving the aforementioned setting value from the information processing device, The aforementioned information processing device is A fourth receiving means for receiving information on the setting items of the electronic device, A first transmission means transmits the arbitrary text information and information about the setting item of the electronic device to a selection means that selects at least one setting item of the electronic device based on arbitrary text information received as input as instructions from the user, A first receiving means that receives the setting item from the selection means, A second transmission means transmits information about the configurable setting values ​​for the setting item received by the first receiving means to a determination means that determines the setting value for executing a process in response to instructions from the user, A second receiving means for receiving the set value from the determination means, The system comprises a fourth transmitting means for transmitting the setting value received by the second receiving means to the electronic device, An electronic device characterized by performing processing based on the set value received by the third receiving means.

[0110] (Composition 3) The electronic device according to configuration 1 or configuration 2, characterized in that the selection means and the decision means are large-scale language models.

[0111] (Composition 4) The first transmitting means transmits a first prompt to cause the electronic device to select at least one setting item, The second transmission means transmits a second prompt to determine a setting value for executing a process in response to instructions from the user. The first prompt includes the arbitrary text information and the information of the setting items of the electronic device. The electronic device according to configuration 3, characterized in that the second prompt includes information on the configurable setting values ​​for the setting item.

[0112] (Composition 5) The electronic device according to configuration 1 or configuration 2, characterized in that the selection options for the setting items in the selection means are all or some of the setting items provided in the electronic device.

[0113] (Composition 6) The electronic device according to configuration 1 or configuration 2, characterized in that the information of the setting items of the electronic device also includes the current setting value of the setting item.

[0114] (Composition 7) The electronic device according to configuration 1 or configuration 2, characterized in that the selection of setting values ​​in the determination means is all or some of the setting values ​​provided in the electronic device.

[0115] (Composition 8) The electronic device is an imaging device having an imaging means, and is characterized in that it performs automatic shooting as a process according to the set value, as described in configuration 1 or configuration 2.

[0116] (Composition 9) The electronic device is a printer, characterized in that it performs a printing process as a process according to the set value, as described in configuration 1 or configuration 2.

[0117] (Composition 10) A first transmission means transmits the arbitrary text information and the information of the electronic device's setting item to a selection means that selects at least one setting item of an electronic device to perform processing according to a setting value based on arbitrary text information received as input as instructions from the user, A first receiving means that receives the setting item from the selection means, A second transmission means transmits information about the configurable setting values ​​for the setting item received by the first receiving means to a determination means that determines the setting value for executing a process in response to instructions from the user, A second receiving means for receiving the set value from the determination means, An information processing apparatus characterized by having a fourth transmitting means for transmitting the setting value received by the second receiving means to the electronic device.

[0118] (System 1) A system comprising electronic and information processing equipment that performs processing according to set values, A first transmission means transmits the arbitrary text information and information about the setting item of the electronic device to a selection means that selects at least one setting item of the electronic device based on arbitrary text information received as input as instructions from the user, A first receiving means that receives the setting item from the selection means, A second transmission means transmits information about the configurable setting values ​​for the setting item received by the first receiving means to a determination means that determines the setting value for executing a process in response to instructions from the user, A second receiving means for receiving the set value from the determination means, A system characterized by having a control means that controls the system to perform processing based on the set value received by the second receiving means.

[0119] (System 2) A system comprising electronic and information processing equipment that performs processing according to set values, The aforementioned electronic device is A third transmission means for transmitting information of the setting items of the electronic device to an information processing device, It has a third receiving means for receiving the aforementioned setting value from the information processing device, The aforementioned information processing device is A fourth receiving means for receiving information on the setting items of the electronic device, A first transmission means transmits the arbitrary text information and information about the setting item of the electronic device to a selection means that selects at least one setting item of the electronic device based on arbitrary text information received as input as instructions from the user, A first receiving means that receives the setting item from the selection means, A second transmission means transmits information about the configurable setting values ​​for the setting item received by the first receiving means to a determination means that determines the setting value for executing a process in response to instructions from the user, A second receiving means for receiving the set value from the determination means, The system comprises a fourth transmitting means for transmitting the setting value received by the second receiving means to the electronic device, The electronic device is characterized by performing processing based on the set value received by the third receiving means.

[0120] (Method 1) A control method for electronic equipment that performs processing according to a set value, A first transmission step involves transmitting the arbitrary text information and information about the setting item of the electronic device to a selection means that selects at least one setting item of the electronic device based on arbitrary text information received as input as instructions from the user, A first receiving step of receiving the setting item from the selection means, A second transmission step transmits information about the configurable setting values ​​for the setting item received in the first receiving step to a determination means that determines setting values ​​for executing processing in response to instructions from the user, A second receiving step of receiving the set value from the determination means, A control method for electronic equipment, characterized by comprising: a control step that controls the device to perform processing based on the set value received in the second receiving step.

[0121] (Method 2) A control method for electronic equipment that performs processing according to a set value, A third transmission step involves transmitting information about the settings of the electronic device to an information processing device, The system includes a third receiving step of receiving the aforementioned setting value from the information processing device, The aforementioned information processing device is A fourth receiving means for receiving information on the setting items of the electronic device, A first transmission means transmits the arbitrary text information and information about the setting item of the electronic device to a selection means that selects at least one setting item of the electronic device based on arbitrary text information received as input as instructions from the user, A first receiving means that receives the setting item from the selection means, A second transmission means transmits information about the configurable setting values ​​for the setting item received by the first receiving means to a determination means that determines the setting value for executing a process in response to instructions from the user, A second receiving means for receiving the set value from the determination means, The system comprises a fourth transmitting means for transmitting the setting value received by the second receiving means to the electronic device, A method for controlling an electronic device, characterized by performing processing based on the set value received in the third receiving step.

[0122] (Method 3) A first transmission step involves transmitting the arbitrary text information and information about the electronic device's setting item to a selection means that selects at least one setting item for an electronic device to perform processing according to a setting value based on arbitrary text information received as input as instructions from the user, A first receiving step of receiving the setting item from the selection means, A second transmission step transmits information about the configurable setting values ​​for the setting item received in the first receiving step to a determination means that determines setting values ​​for executing processing in response to instructions from the user, A second receiving step of receiving the set value from the determination means, A control method for an information processing device, characterized by comprising: a fourth transmission step of transmitting the set value received in the second reception step to the electronic device.

[0123] (Program 1) A program to cause the computer to function as each of the electronic devices described in Configuration 1.

[0124] (Program 2) A program for causing a computer to function as each of the information processing devices described in configuration 10.

[0125] (Storage medium 1) A computer-readable storage medium containing programs that cause the computer to function as each of the electronic devices described in Configuration 1.

[0126] (Storage medium 2) A computer-readable storage medium containing programs for causing the computer to function as each of the information processing devices described in configuration 10.

Claims

1. An electronic device that performs processing according to a set value, A first transmission means transmits the arbitrary text information and information about the setting item of the electronic device to a selection means that selects at least one setting item of the electronic device based on arbitrary text information received as input as instructions from the user, A first receiving means that receives the setting item from the selection means, A second transmission means transmits information about the configurable setting values ​​for the setting item received by the first receiving means to a determination means that determines setting values ​​for executing processing in response to instructions from the user, A second receiving means that receives the set value from the determination means, An electronic device characterized by having a control means for controlling the device to perform processing based on the set value received by the second receiving means.

2. An electronic device that performs processing according to a set value, A third transmission means that transmits information of the setting items of the electronic device to an information processing device, It has a third receiving means for receiving the aforementioned setting value from the information processing device, The aforementioned information processing device is A fourth receiving means for receiving information on the setting items of the electronic device, A first transmission means transmits the arbitrary text information and information about the setting item of the electronic device to a selection means that selects at least one setting item of the electronic device based on arbitrary text information received as input as instructions from the user, A first receiving means that receives the setting item from the selection means, A second transmission means transmits information about the configurable setting values ​​for the setting item received by the first receiving means to a determination means that determines setting values ​​for executing processing in response to instructions from the user, A second receiving means that receives the set value from the determination means, The system comprises a fourth transmitting means for transmitting the set value received by the second receiving means to the electronic device, An electronic device characterized by performing processing based on the set value received by the third receiving means.

3. The electronic device according to claim 1 or 2, characterized in that the selection means and the decision means are large-scale language models.

4. The first transmitting means transmits a first prompt to cause the electronic device to select at least one setting item, The second transmission means transmits a second prompt to determine a setting value for executing a process in response to instructions from the user. The first prompt includes the arbitrary text information and the information of the setting items of the electronic device. The electronic device according to claim 3, characterized in that the second prompt includes information on the configurable setting values ​​for the setting item.

5. The electronic device according to claim 1 or 2, characterized in that the selection options for the setting items in the selection means are all or some of the setting items provided in the electronic device.

6. The electronic device according to claim 1 or 2, characterized in that the information of the setting items of the electronic device also includes the current setting value of the setting item.

7. The electronic device according to claim 1 or 2, characterized in that the selection of setting values ​​in the determination means is all or some of the setting values ​​provided in the electronic device.

8. The electronic device is an imaging device having an imaging means, characterized in that it performs imaging as a process according to the set value, as described in claim 1 or 2.

9. The electronic device is a printer, characterized in that it performs printing as a process according to the set value, as described in claim 1 or 2.

10. A first transmission means transmits the arbitrary text information and the information of the electronic device's setting item to a selection means that selects at least one setting item of an electronic device to perform processing according to a setting value based on arbitrary text information received as input as instructions from the user, A first receiving means that receives the setting item from the selection means, A second transmission means transmits information about the configurable setting values ​​for the setting item received by the first receiving means to a determination means that determines setting values ​​for executing processing in response to instructions from the user, A second receiving means that receives the set value from the determination means, An information processing apparatus characterized by having a fourth transmitting means for transmitting the setting value received by the second receiving means to the electronic device.

11. A system comprising electronic and information processing equipment that performs processing according to set values, A first transmission means transmits the arbitrary text information and information about the setting item of the electronic device to a selection means that selects at least one setting item of the electronic device based on arbitrary text information received as input as instructions from the user, A first receiving means that receives the setting item from the selection means, A second transmission means transmits information about the configurable setting values ​​for the setting item received by the first receiving means to a determination means that determines setting values ​​for executing processing in response to instructions from the user, A second receiving means that receives the set value from the determination means, A system characterized by having a control means that controls the system to perform processing based on the set value received by the second receiving means.

12. A system comprising electronic and information processing equipment that performs processing according to set values, The aforementioned electronic device is A third transmission means that transmits information of the setting items of the electronic device to an information processing device, It has a third receiving means for receiving the aforementioned setting value from the information processing device, The aforementioned information processing device is A fourth receiving means for receiving information on the setting items of the electronic device, A first transmission means transmits the arbitrary text information and information about the setting item of the electronic device to a selection means that selects at least one setting item of the electronic device based on arbitrary text information received as input as instructions from the user, A first receiving means that receives the setting item from the selection means, A second transmission means transmits information about the configurable setting values ​​for the setting item received by the first receiving means to a determination means that determines setting values ​​for executing processing in response to instructions from the user, A second receiving means that receives the set value from the determination means, The system comprises a fourth transmitting means for transmitting the set value received by the second receiving means to the electronic device, The electronic device is characterized by performing processing based on the set value received by the third receiving means.

13. A control method for electronic equipment that performs processing according to a set value, A first transmission step involves transmitting the arbitrary text information and information about the setting item of the electronic device to a selection means that selects at least one setting item of the electronic device based on arbitrary text information received as input as instructions from the user, A first receiving step of receiving the setting item from the selection means, A second transmission step transmits information about the configurable setting values ​​for the setting item received in the first reception step to a determination means that determines setting values ​​for executing processing in response to instructions from the user, A second receiving step of receiving the set value from the determination means, A method for controlling an electronic device, comprising: a control step that controls the device to perform processing based on the set value received in the second receiving step.

14. A control method for electronic equipment that performs processing according to a set value, A third transmission step involves transmitting information about the settings of the electronic device to an information processing device, The system includes a third receiving step of receiving the aforementioned setting value from the information processing device, The aforementioned information processing device is A fourth receiving means for receiving information on the setting items of the electronic device, A first transmission means transmits the arbitrary text information and information about the setting item of the electronic device to a selection means that selects at least one setting item of the electronic device based on arbitrary text information received as input as instructions from the user, A first receiving means that receives the setting item from the selection means, A second transmission means transmits information about the configurable setting values ​​for the setting item received by the first receiving means to a determination means that determines setting values ​​for executing processing in response to instructions from the user, A second receiving means that receives the set value from the determination means, The system comprises a fourth transmitting means for transmitting the set value received by the second receiving means to the electronic device, A method for controlling an electronic device, characterized by performing processing based on the set value received in the third receiving step.

15. A first transmission step involves transmitting the arbitrary text information and information about the electronic device's setting item to a selection means that selects at least one setting item for an electronic device to perform processing according to a setting value based on arbitrary text information received as input as instructions from the user, A first receiving step of receiving the setting item from the selection means, A second transmission step transmits information about the configurable setting values ​​for the setting item received in the first reception step to a determination means that determines setting values ​​for executing processing in response to instructions from the user, A second receiving step of receiving the set value from the determination means, A control method for an information processing device, characterized by comprising: a fourth transmission step of transmitting the set value received in the second reception step to the electronic device.

16. A program for causing a computer to function as one of the means of the electronic device described in claim 1.

17. A program for causing a computer to function as each of the means of the information processing apparatus described in claim 10.

18. A computer-readable storage medium storing a program for causing the computer to function as one of the means of the electronic device described in claim 1.

19. A computer-readable storage medium storing a program for causing the computer to function as each of the means of the information processing apparatus described in claim 10.

Images