Electronic device, information processing apparatus, system, control method for electronic device, control method for information processing apparatus, and non-transitory computer readable medium

Abstract

The present application provides an electronic device, an information processing apparatus, a system, a control method of an electronic device, a control method of an information processing apparatus, and a non-transitory computer readable medium. The electronic device performs processing according to a setting value, and includes one or more processors that execute a program stored in a memory and function as: a first transmission unit that transmits arbitrary text information and information related to a setting item to a selection unit that selects at least one setting item of the electronic device based on arbitrary text information received as an instruction input by a user; a first reception unit that receives the setting item from the selection unit; a second transmission unit that transmits information related to a setting value settable in the setting item to a determination unit that determines the setting value used to perform the processing according to the instruction; a second reception unit that receives the setting value from the determination unit; and a control unit that controls to perform the processing based on the setting value.

Description

Technical Field

[0001] This disclosure relates to electronic devices, information processing apparatuses, systems, control methods for electronic devices, control methods for information processing apparatuses, and non-volatile computer-readable media that perform automatic processing based on instructions from users. Background Technology

[0002] In recent years, systems that automatically begin processing in response to user voice input have reached the stage of practical application. This technology significantly reduces the need for manual operation and enables more intuitive and rapid control of electronic devices. Japanese Patent Application Publication No. 2022-111133 discloses a camera instruction method in which, when a user utters a keyword indicating the start of camera recording (e.g., "take a picture"), the voice is recognized by an audio processing unit and used as a trigger for camera recording.

[0003] In the technology described in Japanese Patent Application Publication No. 2022-111133, voice commands are limited to pre-registered phrases, and users need to memorize and use specific phrases. Summary of the Invention

[0004] This disclosure was made in view of the above circumstances and is intended to provide an electronic device that can control processing according to any expressed instructions in automatic processing based on user instructions.

[0005] According to an aspect of this disclosure, an electronic device is provided that performs processing based on setting values. The electronic device includes one or more processors that execute programs stored in a memory, thereby serving as: a first transmitting unit, a first receiving unit, a second transmitting unit, a second receiving unit, and a control unit. The first transmitting unit is configured to transmit arbitrary text information received as a user input instruction and information relating to at least one setting item of the electronic device to a selection unit, the selection unit being configured to select at least one setting item of the electronic device based on the arbitrary text information. The first receiving unit is configured to receive the setting item from the selection unit. The second transmitting unit is configured to transmit information relating to a settable setting value among the setting items received by the first receiving unit to a determining unit, the determining unit being configured to determine a setting value for performing processing based on an instruction from the user. The second receiving unit is configured to receive the setting value from the determining unit. The control unit is configured to perform control to perform processing based on the setting value received by the second receiving unit.

[0006] Other features of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. The following description of the embodiments is by way of example. Attached Figure Description

[0007] Figure 1An example is provided of a process for determining camera settings using a generative artificial intelligence (AI) service in a camera according to a first embodiment.

[0008] Figure 2 The configuration of the camera according to the first embodiment is illustrated.

[0009] Figure 3 An example of a screen display of a smartphone application used in the first embodiment is shown.

[0010] Figure 4A and Figure 4B Examples of prompts for selecting camera settings by the generative AI service in the first embodiment are shown respectively.

[0011] Figure 5 This is a flowchart of the process for interpreting messages sent from the generative AI service in the first embodiment.

[0012] Figure 6A and Figure 6B Examples of prompts for the generative AI service to select camera setting values ​​are shown in the first embodiment.

[0013] Figure 7 An example is provided of a process for determining printer settings using a generative AI service in a printer according to the second embodiment.

[0014] Figure 8 An example screen of a printer driver personal computer (PC) application used in the second embodiment is shown.

[0015] Figure 9A and Figure 9B Examples of prompts for selecting printer settings by the generative AI service in the second embodiment are shown respectively.

[0016] Figure 10A and Figure 10B Examples of prompts for the generative AI service to select printer settings values ​​are shown in the second embodiment. Detailed Implementation

[0017] The present disclosure will now be described in detail with reference to the accompanying drawings and exemplary embodiments thereof.

[0018] The following embodiments do not limit the scope of the claims. Although multiple features are described in the various embodiments, not all features are essential to this disclosure. Furthermore, multiple features can be combined arbitrarily. Moreover, in the drawings, the same or similar parts are given the same reference numerals, and redundant descriptions are omitted.

[0019] The following embodiments of this disclosure relate to a gimbal camera (first embodiment) and a printer (second embodiment) as examples of electronic devices. However, electronic devices are not limited to those described above. Other examples of electronic devices include home appliances (e.g., refrigerators and microwave ovens) and office equipment (e.g., multifunction devices). The first embodiment relates to a gimbal camera as an example of a camera device, but it may also relate to another camera device. Examples include digital cameras, camcorders, smartphones, tablets, wearable cameras, smartwatches, smart glasses, webcams, security cameras, gaming devices, robots, drones, and dashcams. Examples of printers include inkjet printers, laser printers, 3D printers, sublimation printers, portable printers, and large-format printers.

[0020] First Embodiment

[0021] This embodiment relates to an example of processing such that the desired shooting is performed even when the user (photographer) gives the camera shooting instructions that include verbal expressions, such as "shoot with Mr. / Ms. A in the center", "keep shooting the children", or "take photos continuously for about five minutes".

[0022] In this embodiment, the method for fine-grained control of a device based on arbitrary natural language expressions involves enabling a generative artificial intelligence (AI) service to interpret the input natural language and convert it into device control commands. In this case, when an arbitrary natural language-based instruction is to be sent to the generative AI service, device-prepared settings and options for the settings values ​​in those settings are added to a prompt, which is then provided to the generative AI service, thereby enabling fine-grained and appropriate device settings. However, some generative AI services are chargeable. In particular, when using application programming interfaces (APIs), some employ pay-as-you-go billing schemes that determine usage fees based on the number of characters or words in the text. Therefore, it is also considered to suppress costs by reducing the number of text characters in the prompts sent to the generative AI service. The generative AI service is configured using a large language model (LLM). A large language model is a deep learning model consisting of an artificial neural network with a large number of parameters that generates and outputs appropriate responses to natural language-based instructions (prompts).

[0023] Camera control based on natural language commands

[0024] Figure 1The illustration depicts a process of controlling camera 102 based on natural language commands spoken by user 101. In this embodiment, smartphone application 103 serves as an information processing device for receiving and interpreting the voice input of user 101. If camera 102 is equipped with audio receiving capabilities as well as voice analysis and text conversion capabilities, a configuration without smartphone application 103 is also possible. Reference numeral 104 denotes a text-generating AI service, which is assumed to be an internet-based generative AI service (e.g., ChatGPT).

[0025] The attached diagram shows a camera control voice command based on voice input from user 101. An example of this command is "Take a picture of this dish to make it look delicious." The camera control voice command M111 is first received by the smartphone application 103. The smartphone application 103 interprets the received voice and converts it into text data. There are several methods for converting speech to text. In the case of an Android application, a speech recognition library can be used. Web services that convert speech data into text strings can also be used. For example, speech-to-text (https: / / cloud.google.com / speech-to-text) provided by Google LLC can be used.

[0026] Reference numeral M112 indicates a camera settings list request message, used to send a request for a list of camera settings from smartphone application 103 to camera 102. Upon receiving a camera control voice command M111 from user 101, smartphone application 103 sends the camera settings list request message M112 to camera 102.

[0027] Reference numeral M113 indicates a response to a camera settings list request made by camera 102, as a reply to the camera settings list request message M112. Details of the response include a list of camera settings to be determined by generative AI service 104, and the current setting value for each camera setting. The reason for including the current setting value in the response is to accommodate a request from user 101 to correspondingly change the current camera settings, such as "take a slightly brighter photo than the previous one." The camera settings included in the camera settings list request response M113 can be changed based on the camera's status or subject detection status. For example, if user 101 has already manually set the shutter speed, that manual setting can be prioritized, and the shutter speed can be excluded from the camera settings list.

[0028] Reference numeral M114 indicates a camera setting selection prompt, used to enable the text-generating AI service 104 to select at least one camera setting from the smartphone application 103. Upon receiving a camera setting list request response M113 from the camera 102, the smartphone application 103 creates the camera setting selection prompt M114 and sends it to the text-generating AI service 104. Figure 4A As shown, the camera settings selection prompt M114 includes details of the camera control voice command M111 and the camera settings list request response M113.

[0029] In the attached diagram, reference numeral M115 indicates the camera setting selection result (first-stage response) from the text-generating AI service 104, which has already received the camera setting selection prompt M114. For example... Figure 4B As shown, the text-generating AI service 104 returns the selection results for "Which settings should be changed to take photos that meet the photographer's requirements?"

[0030] Reference numeral M116 indicates a camera setting value list request message, which is sent by smartphone application 103 to camera 102 after receiving camera setting item selection result M115. Smartphone application 103 extracts the camera setting items to be changed from the details of camera setting item selection result M115. Then, as a request for a list of settable camera setting values ​​from the extracted camera setting items, camera setting value list request message M116 is sent to camera 102.

[0031] Reference numeral M117 indicates a response to a camera settings list request sent from camera 102 to smartphone application 103. In response to a camera settings list request message M116 received from smartphone application 103, camera 102 creates a list of settings that can be set in the target camera settings and sends the list of settings to smartphone application 103.

[0032] Reference numeral M118 indicates a camera setting value determination prompt provided by the smartphone application 103 to the text-generating AI service 104 after receiving a camera setting value list request response M117, so that the text-generating AI service 104 can determine the appropriate setting value from the setting value list. The camera setting value determination prompt M118 includes a camera control voice command M111, such that the setting value is the value desired by the user 101.

[0033] The attached figure shows that the camera setting selection result (second-stage response) comes from the text-generating AI service 104, which has already received the camera setting value confirmation prompt M118.

[0034] Reference numeral M120 indicates a camera setting change message, which enables smartphone application 103 to set camera settings in camera 102. These camera settings are determined based on camera setting selection result M119 from text-generating AI service 104. Upon receiving the camera setting change message M120, camera 102 changes the camera settings and performs a shooting operation.

[0035] As described above, the response from the text-generating AI service 104 in the first stage includes several camera settings, while the response from the text-generating AI service 104 in the second stage includes settings values ​​from a more narrowly defined set of settings than in the first stage. Therefore, an appropriate settings response can be obtained, and the amount of communication text (token quantity) with the text-generating AI service 104 can be suppressed.

[0036] Camera configuration

[0037] Figure 2 This is a block diagram illustrating the internal configuration of the camera 102 according to this embodiment.

[0038] The lens barrel 200 has a camera optical system and a camera element that acquires image data based on the light flux from the camera optical system, and is attached to the fixed part (not shown) of the camera 102 via a rotatably driven rotation mechanism.

[0039] Lens unit 201 includes a zoom unit and a focusing unit. The zoom unit includes a zoom lens that performs variable magnification. The focusing unit includes a lens that performs focusing. Lens unit 201 is driven and controlled by lens drive unit 210.

[0040] The imaging unit 202 includes an imaging element. The imaging element receives the light flux incident through each lens group and generates charge information as analog image data based on the amount of light flux. The analog image data is output to the image processing unit 212.

[0041] The control box 204 includes, for example, a camera lens assembly included in the lens barrel 200, and a control microcomputer for controlling the tilt rotation unit 205 and the pan rotation unit 206. In this embodiment, the control box 204 is disposed within the mounting portion of the camera 102, such that the control box 204 remains fixed even when the lens barrel 200 is performing panning and tilting operations.

[0042] The lens barrel drive unit 211 drives the tilt rotation unit 205 and the panning rotation unit 206, thereby rotating the lens barrel 200 in the tilt and panning directions. The lens barrel drive unit 211 is driven and controlled by the control unit 217.

[0043] By using an aperture control unit (not shown), a sensor gain control unit, and a shutter control unit, the camera 102 performs exposure control to ensure that the subject has appropriate brightness.

[0044] The image processing unit 212 converts the analog image data input from the camera unit 202 into digital image data via analog-to-digital (A / D) conversion. The image processing unit 212 applies image processing techniques such as distortion correction, white balance adjustment, and color interpolation to the digital image data and outputs the processed digital image data. The digital image data output from the image processing unit 212 is converted into a recordable format such as JPEG by the recording unit 213 and sent to the random access memory (RAM) 219 and the recording medium 214.

[0045] Recording unit 213 records, for example, compressed image signals, compressed audio information, and other camera-related control data generated by image processing unit 212 and audio processing unit 215 onto recording medium 214. If the audio signal is not compressed and encoded, control unit 217 sends the audio signal generated by audio processing unit 215 and the compressed image signal generated by image processing unit 212 to recording unit 213, and causes recording unit 213 to record the signals onto recording medium 214.

[0046] Recording medium 214 is contained within camera 102, but may also be a removable recording medium. Various data, such as compressed image signals, compressed audio signals, and audio signals generated by camera 102, can be recorded on recording medium 214. Therefore, a medium with a capacity greater than read-only memory (ROM) 220 is typically used as recording medium 214. For example, recording medium 214 can be any type such as hard disk, optical disk, magneto-optical disk, recordable optical disk (CD-R), recordable digital multifunction optical disk (DVD-R), magnetic tape, non-volatile semiconductor memory, or flash memory.

[0047] The audio processing unit 215 performs audio-related processing, such as optimizing the input digital audio signal. The audio signal processed by the audio processing unit 215 is sent to the RAM 219 by the control unit 217.

[0048] The display unit 216 has, for example, a function for outputting visually identifiable information, such as a liquid crystal display (LCD) or a light-emitting diode (LED) display.

[0049] The control unit 217 includes, for example, a central processing unit (CPU) (e.g., a microprocessor unit (MPU)), memory (e.g., dynamic random access memory (DRAM) or static random access memory (SRAM)), and non-volatile memory (electrically erasable programmable ROM (EEPROM)). The control unit 217 executes various processes (programs) to control the various blocks of the camera 102 and control the data transmission between the blocks.

[0050] Communication unit 218 performs communication with external devices (such as camera 102 and smartphone application 103), and sends and receives data such as audio signals, image signals, compressed audio signals, compressed image signals, and text messages. When camera 102 detects an abnormal state, communication unit 218 sends information to the respective external devices to notify them of the internal state of the camera device, such as error messages. Communication unit 218 includes, for example, wireless communication modules, such as infrared communication modules, Bluetooth communication modules, wireless local area network (LAN) communication networks, wireless universal serial bus (USB) and / or global positioning system (GPS) receivers.

[0051] RAM 219 temporarily stores the image signals and audio signals obtained by the image processing unit 212 and the audio processing unit 215.

[0052] ROM 220 is an electrically erasable and recordable memory, and stores, for example, control constants and programs for the operation of control unit 217.

[0053] The operation unit 221 is an input device that receives various operations performed by the user 101. Examples of usable operation units 221 are touchscreens or physical buttons. For example, a touchscreen may be disposed on the display surface of the display unit 216 and integrated with the display unit 216. The operation unit 221 and the display unit 216 may or may not be detachable from the camera 102. The operation unit 221 may be implemented as one of the applications of a general computing device such as a smartphone.

[0054] The audio input unit 222 acquires audio signals from the microphone set in the camera 102 around the camera 102, performs analog-to-digital conversion on the audio signals, and sends the audio signals to the audio processing unit 215.

[0055] The subject detection unit 223 detects subjects included in the captured image and determines the attributes of the subjects. The subject detection unit 223 detects the face and body of the subject. During the face detection process, a pattern for determining the face of the subject is preset, and the area that matches the pattern included in the captured image is detected as the face image of the subject.

[0056] In addition, a reliability level indicating the certainty of the subject's face is simultaneously calculated. The reliability level is calculated based on, for example, the size of the facial region within the image and the degree of matching with the facial pattern. Similarly, subjects matching pre-registered patterns can be identified for subject recognition.

[0057] There are also methods for extracting feature subjects from captured images using histograms (e.g., hue and saturation). In this case, for an image of a subject captured within the camera's viewpoint, the distribution derived from the histogram (e.g., hue and saturation) is divided into multiple intervals, and classification processing is performed on the captured images for each interval. For example, a histogram of multiple color components is created for the captured image, and the image is divided based on the peak distribution range of the histogram. The captured images are then classified into regions belonging to combinations of the same components, and image regions of the subject are identified. By calculating the evaluation value of each image region of the identified subject, the image region of the subject with the highest evaluation value is determined as the main subject region. Using the above method, information about each subject can be obtained from the captured image information.

[0058] The subject detection unit 223 also performs attribute estimation on the detected subject. Attribute estimation involves estimating attributes using a predetermined deterministic expression, based on, for example, edge and contour information about the eyes and mouth in the detected facial region. In another embodiment, the method and content are not defined, as is the case when using machine learning. In this embodiment, the type of the subject is estimated, i.e., whether the subject is a human, a cat, or another biological category. The attributes to be estimated can be other than those described above, and may include, for example, race, facial orientation, facial shape, organ and hair color, and whether items are worn (e.g., mask, glasses, sunglasses, eye shield, headscarf, or collar).

[0059] Image processing unit 212 and audio processing unit 215 read image signals and audio signals temporarily stored in RAM 219, and encode the image signals and audio signals respectively to generate compressed image signals and compressed audio signals.

[0060] Screenshot of smartphone application 103

[0061] Figure 3 An example of a screen display for smartphone application 103 is shown.

[0062] The display 301 of the smartphone with smartphone application 103 installed displays items 302 to 305, which are to be described below.

[0063] Item 302 is the audio input button. Clicking the audio input button 302 changes the current state to audio input receiving state. Clicking the audio input button 302 again terminates the audio input receiving state. Voice input from the beginning to the end of the audio input receiving state is considered a single camera command.

[0064] Item 303 indicates the camera instruction voice text obtained as a result of converting the voice of the camera instruction input to the smartphone application 103 into text.

[0065] Item 304 indicates an image preview taken by camera 102 based on the camera command given by user 101. The most recently captured image from camera 102 is acquired and displayed on the screen.

[0066] Item 305 indicates the response message from the smartphone application 103 when a camera recording command is received. Item 305 indicating the response message can display the camera settings used during recording. In this case, user 101 can check the camera settings displayed in item 305 indicating the response message, manually change the camera settings, and take a picture.

[0067] Communication between camera 102 and smartphone application 103

[0068] The following description pertains to communication between camera 102 and smartphone application 103. The communication method used is the HTTP protocol. The communication unit 218 of camera 102 has the function of receiving and interpreting HTTP protocol data.

[0069] The exchange between camera 102 and smartphone application 103 consists of the following five items.

[0070] 1. Camera settings list request message M112

[0071] 2. Camera settings list request response M113, which is the response to M112.

[0072] 3. Camera Settings List Request Message M116

[0073] 4. Camera settings list request response M117, which is the response to M116.

[0074] 5. Camera settings changed message M120

[0075] First, an example of an HTTP request for the camera settings list request message M112 is as follows: GET http: / / [IP address]:[port number] / cameraapi / camerasettinglist HTTP / 1.1

[0076] Upon receiving this HTTP request, camera 102 lists the camera settings to be included in the response message. The listed camera settings may be one or more settings present in camera 102, not all settings. As a specification of camera 102, the range of camera settings to be included in the response can be narrowed down in advance. The range of camera settings to be included in the response can be narrowed down based on the state detected by camera 102. For example, if no face is detected in subject detection, face-related settings (e.g., settings for prioritizing autofocus on faces and / or settings for increasing face exposure) are excluded.

[0077] When determining the camera settings to be included in the response, camera 102 creates a list of camera settings as a response request M113. In this embodiment, the camera settings to be included in the response are "shutter speed", "aperture", "ISO", "exposure correction", "white balance", "continuous shooting mode", "contrast", "color filter effect", "HDR shooting", and "flash mode". The response uses JSON as the data format and has the following content.

[0078]

[0079]

[0080] The following is an example of an HTTP request for a list of camera settings message M116.

[0081] Shutter speed

[0082] GET http: / / [IP address]:[port number] / cameraapi / camerasetting / shutterspeed HTTP / 1.1

[0083] White Balance

[0084] GET http: / / [IP address]:[port number] / cameraapi / camerasetting / wbHTTP / 1.1

[0085] In this embodiment, a Uniform Resource Locator (URL) for an HTTP request is prepared for each camera setting item. Based on the camera setting item selection result M115 received from the text-generating AI service 104, the smartphone application 103 extracts the camera setting items to be changed and sends an HTTP request to the camera 102 using the URL corresponding to the extracted camera setting item.

[0086] Upon receiving an HTTP request, camera 102 lists the camera setting values ​​to be included in the response message. Similar to the response M113 request to create a list of camera setting items, the camera setting values ​​to be listed can be one or more settings present in camera 102, not all settings. For example, in the case of aperture, even if the camera is capable of stopping down the aperture to a size smaller than f / 32, if the aperture is stopped down to a size smaller than f / 32, it may sometimes be impossible to achieve proper exposure, even using the maximum shutter speed and ISO sensitivity. In this case, the camera setting included in the response is limited to f / 32.

[0087] Once the camera settings to be included in the response are determined, camera 102 creates a camera setting value list request response M117. In this embodiment, JSON format is used, and the content is as follows.

[0088] aperture

[0089]

[0090]

[0091] The following is an example of an HTTP request for a camera settings change message from the M120.

[0092] For the URL of the request used for the camera settings value list request message M116, issue a PUT command instead of a GET command.

[0093] Shutter speed

[0094] PUT http: / / [IP address]:[port number] / cameraapi / camerasetting / shutterspeed HTTP / 1.1

[0095] In this embodiment, the camera settings set based on the PUT command use JSON data format, and the content is as follows:

[0096] {

[0097] Value: "1 / 60"

[0098] }

[0099] Example of camera settings selection prompts for M114

[0100] Figure 4AAn example of the camera settings selection prompt M114 described above is illustrated. First, in response to the photographer's request, the text-generating AI service 104 is assigned the role of providing appropriate camera settings, much like a professional photographer would. This response method is limited. Although an upper limit is set on the number of items that can be selected from the settings, this limit is not mandatory. The response format of the text-generating AI service 104 is also specified in detail. Figure 4A In this context, the format is restricted to "Item 1, Item 2, Item 3...". This is because, unless the format is restricted in this way, the text-generating AI service 104 will respond in natural language, making it difficult for the smartphone application 103 to interpret the response programmatically. Despite this... Figure 4A In the example, the camera setting item name is included in the prompt as is, but the camera setting item name can also be replaced with other terms. For example, the setting item name "High Brightness Tone Priority" can be replaced with an expression such as "Reduce White Hollows". Furthermore, multiple camera setting items can be replaced with a combined item. For example, "Color Temperature" and "Contrast" can be combined into an expression such as "Warm Tones in Photos". However, when a camera setting item is set to a different expression, as mentioned above, the expression corresponding to the camera setting value also needs to be changed.

[0101] Figure 4B An example is shown of camera setting selection results M115 sent from text-generating AI service 104 to smartphone application 103. The message includes camera setting items "Aperture," "Exposure Correction," and "Contrast," separated by commas, and is... Figure 4A The prompt specifies the format.

[0102] Message Interpretation Sequence of Smartphone Application 103

[0103] Figure 5 An example is shown of a smartphone application 103 receiving the camera settings selection result M115 and interpreting the sequence of messages.

[0104] In step S501, the received text string of camera setting selection result M115 is separated by commas and stored in a text string array. Figure 4B In the example, "aperture", "exposure correction", and "contrast" are stored in the elements of the text string array.

[0105] Text string array [0] = "Aperture", text string array [1] = "Exposure Correction", text string array [2] = "Contrast"

[0106] Step S502 involves initializing a counter variable i, which is used to check whether each text string stored in the text string array exists in the loop process in the camera settings.

[0107] Step S503 is the starting point for a loop process used to check whether each text string stored in the text string array exists in the camera settings. The loop process is repeated for the number of elements in the text string array. In this embodiment, the loop process is executed three times for i = 0, 1, 2.

[0108] Step S504 involves checking whether each text string in the text string array [i] that serves as a camera setting item is valid. A list of camera settings items is obtained from the camera via the camera setting item list request response M113. It is then checked whether a list of camera settings items exists that matches a text string in the text string array [i].

[0109] In step S505, if the text string in the text string array [i] in step S504 is a camera setting item name, then that text string is added to the setting change item list. The setting change item list is maintained by the smartphone application 103 in the form of, for example, a variable or a database. Based on this information, a camera setting value list request message M116 is created and sent to the camera 102.

[0110] In step S506, the loop counter variable i is incremented by 1.

[0111] In step S507, if the loop counter variable i increases the number of elements in the text string array, the process exits the loop and ends the process used to interpret the camera setting selection result M115.

[0112] The camera setting selection result M115 is generated by the text-generating AI service 104 and may not be in the expected format, may include invalid camera setting items, or may not be an appropriate answer. If the number of elements in the setting change item list is zero, or if the number of elements in the text string array is different from the number of elements in the setting change item list, the camera setting selection prompt M114 can be sent again to the text-generating AI service 104.

[0113] Example of camera setting confirmation prompt for M118

[0114] Figure 6A An example of the camera setting value confirmation prompt M118 described above is provided. Similar to the camera setting selection prompt M114, in response to the photographer's request, the text-generating AI service 104 is assigned the role of providing appropriate camera settings and also specifies the response format. Furthermore, Figure 6A Includes a list of camera setting values ​​from the selected camera settings in the camera settings selection results M115.

[0115] Figure 6BAn example is shown of camera setting value selection result M119 sent from text-generating AI service 104 to smartphone application 103. For example... Figure 6A As indicated in the prompt, combinations of camera settings in the camera settings are represented in the form of "Settings:Settings", such as "Aperture: F2.8", and the settings are separated by commas.

[0116] Overview of the First Embodiment

[0117] Therefore, in this embodiment, even if the instructions from the user are vague spoken expressions, the camera operation expected by the user can still be performed. Furthermore, the query to the text-generating AI service 104 is set in two phases: camera settings items and camera setting values ​​for the selected camera settings items. Therefore, the amount of text can be reduced compared to performing a single query using all combinations of all camera settings items and camera setting values ​​present in camera 102.

[0118] While the above description relates to a speech recognition method using audio input unit 222 and audio processing unit 215 as a function for inputting arbitrary text information commands, other configurations for inputting arbitrary text commands can also be employed. For example, a text string can be directly input using a text input device such as a keyboard. Another alternative configuration could be to receive the text string via, for example, a chat application running on another device (e.g., a smartphone).

[0119] Second Embodiment

[0120] The second embodiment relates to a printer as an example.

[0121] Printer control based on natural language commands

[0122] Now refer to Figure 7 This describes a flow chart of a method for controlling printer 702 based on natural language commands from printer user 701. Reference numeral 703 indicates a printer driver PC application equipped with keyboard-based text input functionality and settings and print command functions for printer 702. Reference numeral 704 indicates a text-generating AI service, assumed to be an internet-based service such as ChatGPT.

[0123] The reference numeral M711 indicates a printer control command given by printer user 701, and is a natural language command entered via keyboard. An example of this command is "Print New Year's Card".

[0124] Reference numeral M712 indicates a printer settings confirmation prompt, used to determine printer settings from the printer driver PC application 703 within the text-generating AI service 704. The printer settings confirmation prompt M712 includes a printer control instruction M711 from the printer user 701 and a list of printer settings. Since the printer driver PC application 703 has already determined the configurable settings in the printer 702, a list of printer settings can be created without querying the printer 702 for settings.

[0125] The reference numeral M713 indicates the printer setting selection result from the text-generated AI service 704, which has received the printer setting confirmation prompt M712.

[0126] Reference numeral M714 indicates a printer setting value determination prompt, used to enable the printer driver PC application 703 to determine the appropriate setting value from the print setting value list in the text-generating AI service 704. The printer setting value determination prompt M714 includes a list of printer setting values ​​for the printer setting item selected based on the printer setting item selection result M713, and the content of the printer control instruction M711. Since the printer driver PC application 703 already knows the setting value to be set in the printer 702, it is able to create the printer setting value list.

[0127] The attached figure shows that the printer setting selection result comes from the text-generating AI service 704, which has received the printer setting value confirmation prompt M714.

[0128] The attached reference numeral M716 indicates a printer print instruction message, used to enable the printer driver PC application 703, which has already received the printer setting value selection result M715 from the text-generating AI service 704, to set the printer settings in the printer 702 and further issue a print instruction to it. Upon receiving the printer print instruction message M716, the printer 702 changes the printer settings according to the instruction and performs the print operation.

[0129] Example of a screen display for a printer driver PC application 703

[0130] Figure 8 This example shows a screenshot of the printer driver PC application 703.

[0131] Items 802 to 808, described later, are displayed on the monitor 801 of the PC, in which the printer driver PC application 703 is opened.

[0132] Item 802 indicates the filename of the print file. This filename may be information indicating what kind of file is being printed by being sent to the text generation AI service 704.

[0133] Item 803 is the print preview screen.

[0134] Item 804 is a chat area with the printer driver PC application 703. The chat area includes a command input section for the printer user 701, a display section for showing input details, and a reply display section from the printer driver PC application 703.

[0135] Item 805 indicates the content entered by printer user 701.

[0136] Item 806 indicates a message from printer driver PC application 703.

[0137] When the printer control command M711 from printer user 701 is insufficient, this section displays the content of the requested additional printer control command M711, such as "Please specify the number of copies". Alternatively, this section displays the determined printer settings and printer setting values ​​so that printer user 701 can check whether the printer is set to the desired settings.

[0138] Item 807 is a text input unit used by printer user 701 for inputting printer control commands M711.

[0139] Item 808 is the print button. When printer user 701 clicks this button, the printer print command message M716 is sent from printer driver PC application 703 to printer 702, thereby initiating the printing operation.

[0140] Example of a confirmation prompt for print settings M712

[0141] Figure 9A An example of the printer settings confirmation prompt M712 described above is illustrated. First, in response to a request from printer user 701, text generation AI service 704 is assigned the role of providing appropriate printer settings. The message assigning the role includes the filename of the file to be printed and information that enables text generation AI service 704 to determine the print settings. The response method is limited to the statement "List the following...". The response format for text generation AI service 704 is also specified in detail. Figure 9A In this context, the format is restricted to "item 1, item 2, item 3...". This is to facilitate the interpretation of the text string representing the printer setting selection result M713 in the printer driver PC application 703, which receives the print setting selection result M713 as a response from the text-generating AI service 704.

[0142] Figure 9B This example illustrates a printer settings selection result M713 sent from a text-generating AI service 704 to a printer driver PC application 703. The settings in printer 702 (such as "Sheet Size," "Sheet Type," and "Color Mode") are comma-separated messages and are... Figure 9A The answer format specified in the document.

[0143] Example of a printer settings confirmation prompt in M714

[0144] Figure 10A An example of the printer setting confirmation prompt M714 described above is provided. Similar to the printer setting confirmation prompt M712, in response to a request from printer user 701, the text-generating AI service 704 is assigned the role of providing appropriate printer settings and the filename of the file to be printed. The response format is specified in detail. Additionally, Figure 10A Includes a list of printer setting values ​​from the printer setting selection results M713.

[0145] Figure 10B An example of a printer setting value selection result M715 sent from the text-generating AI service 704 to the printer driver PC application 703 is shown. The combination of printer setting values ​​in the printer settings items is represented in the form of "Setting Item: Setting Value", such as "Sheet Size: Postcard", and the setting items are separated by commas.

[0146] Overview of the Second Embodiment

[0147] Therefore, in this embodiment, even when the user's instruction is a vague, spoken expression, the user's desired printing operation can still be performed. Furthermore, the query to the text-generating AI service 704 is set up in two stages: for printer settings items and for the printer setting value of the selected printer settings item. Therefore, the amount of text can be reduced compared to performing a single query using all combinations of all printer settings items and printer setting values ​​present in printer 702.

[0148] This embodiment can provide an electronic device that can control processing according to any expressed instructions in user-instruction-based automatic processing.

[0149] Other embodiments

[0150] Embodiments of this disclosure can also be implemented by a computer that reads and executes computer-executable instructions (e.g., one or more programs) recorded on a storage medium (also more fully referred to as a 'non-transitory computer-readable storage medium') to perform the functions of one or more of the above embodiments and / or includes one or more circuits (e.g., application-specific integrated circuits (ASICs)) for performing the functions of one or more of the above embodiments, and can also be implemented by a computer that, for example, reads and executes computer-executable instructions from a storage medium to perform the functions of one or more of the above embodiments and / or controls one or more circuits to perform the functions of one or more of the above embodiments. The computer may include one or more processors (e.g., a central processing unit (CPU), a microprocessor unit (MPU)) and may include a network of separate computers or separate processors that read and execute computer-executable instructions. The computer-executable instructions may be provided to the computer, for example, from a network or a storage medium. The storage medium may include, for example, a hard disk, random access memory (RAM), read-only memory (ROM), storage devices for distributed computing systems, optical discs (such as compressed optical discs (CDs), digital versatile optical discs (DVDs), or Blu-ray discs (BDs). TM One or more of the following: flash memory devices, memory cards, etc.

[0151] While this disclosure has been described with reference to exemplary embodiments, it should be understood that this disclosure is not limited to the disclosed exemplary embodiments. The scope of the claims should be interpreted as broadly as possible to cover all such variations and equivalent structures and functions.

Claims

1. An electronic device that performs processing according to a set value, the electronic device comprising: One or more processors that execute programs stored in memory, thereby serving as: A first sending unit is configured to send arbitrary text information received as a user input instruction and information relating to at least one setting of an electronic device to a selection unit, the selection unit being configured to select at least one setting of the electronic device based on the arbitrary text information. A first receiving unit is configured to receive settings from the selection unit; The second sending unit is configured to send information related to the settable setting value in the setting item received by the first receiving unit to the determining unit, the determining unit being configured to determine the setting value for performing the processing based on an instruction from the user; A second receiving unit is configured to receive a setting value from the determining unit; as well as The control unit is configured to perform control to execute the processing based on the setting values ​​received by the second receiving unit.

2. The electronic device according to claim 1, wherein, The selection unit and the determination unit are characterized by a large language model.

3. The electronic device according to claim 2, in, The first sending unit sends a first prompt, which is used to select at least one setting item of the electronic device. The second sending unit sends a second prompt, which is used to determine the setting value for performing the process according to the instruction from the user. The first prompt includes the arbitrary text information and information related to the settings of the electronic device, and The second prompt includes information related to the settings that can be configured in the settings.

4. The electronic device according to claim 1, wherein, At least one option for the settings in the selection unit includes all or one or more settings contained in the electronic device.

5. The electronic device according to claim 1, wherein, Information relating to the settings of the electronic device includes the current settings values ​​in the settings.

6. The electronic device according to claim 1, wherein, At least one option for the setting value in the determining unit includes all or one or more setting values ​​contained in the electronic device.

7. The electronic device according to claim 1, wherein, The electronic device is a camera device, which has a camera unit and performs shooting as the process according to set values.

8. The electronic device according to claim 1, wherein, The electronic device is a printer, which performs printing according to set values ​​as part of the process.

9. An electronic device that performs processing according to a set value, the electronic device comprising: One or more processors that execute programs stored in memory, thereby serving as: The third transmitting unit is configured to transmit information related to the settings of the electronic device to the information processing device; as well as The third receiving unit is configured to receive setting values ​​from the information processing device. The information processing device includes: The fourth receiving unit is configured to receive information relating to settings of the electronic device; A first sending unit is configured to send arbitrary text information received as a user input instruction and information relating to at least one setting item of the electronic device to a selection unit, the selection unit being configured to select at least one setting item of the electronic device based on the arbitrary text information. A first receiving unit is configured to receive settings from the selection unit; The second sending unit is configured to send information related to the settable setting value in the setting item received by the first receiving unit to the determining unit, the determining unit being configured to determine the setting value for performing the processing based on an instruction from the user; A second receiving unit is configured to receive a setting value from the determining unit; and A fourth transmitting unit is configured to transmit the setting value received by the second receiving unit to the electronic device, and The processing is performed based on the setting value received by the third receiving unit.

10. An information processing apparatus, the information processing apparatus comprising: One or more processors that execute programs stored in memory, thereby serving as: A first sending unit is configured to send arbitrary text information received as a user input instruction and information relating to at least one setting item of an electronic device that performs processing according to a setting value to a selection unit, the selection unit being configured to select at least one setting item of the electronic device based on the arbitrary text information. A first receiving unit is configured to receive settings from the selection unit; The second sending unit is configured to send information related to the settable setting value in the setting item received by the first receiving unit to the determining unit, the determining unit being configured to determine the setting value for performing the processing based on an instruction from the user; A second receiving unit is configured to receive a setting value from the determining unit; as well as A fourth transmitting unit is configured to transmit the setting values ​​received by the second receiving unit to the electronic device.

11. A system comprising an electronic device and an information processing apparatus, wherein the electronic device performs processing according to preset values, the system comprising: A first sending unit is configured to send arbitrary text information received as a user input instruction and information relating to at least one setting item of the electronic device to a selection unit, the selection unit being configured to select at least one setting item of the electronic device based on the arbitrary text information. A first receiving unit is configured to receive settings from the selection unit; The second sending unit is configured to send information related to the settable setting value in the setting item received by the first receiving unit to the determining unit, the determining unit being configured to determine the setting value for performing the processing based on an instruction from the user; A second receiving unit is configured to receive a setting value from the determining unit; as well as The control unit is configured to perform control to execute the processing based on the setting values ​​received by the second receiving unit.

12. A system comprising an electronic device and an information processing apparatus, wherein the electronic device performs processing according to preset values. in, The electronic device includes: A third transmitting unit is configured to transmit information relating to settings of the electronic device to the information processing device; and The third receiving unit is configured to receive setting values ​​from the information processing device. The information processing device includes: The fourth receiving unit is configured to receive information relating to settings of the electronic device; A first sending unit is configured to send arbitrary text information received as a user input instruction and information relating to at least one setting item of the electronic device to a selection unit, the selection unit being configured to select at least one setting item of the electronic device based on the arbitrary text information. A first receiving unit is configured to receive settings from the selection unit; The second sending unit is configured to send information related to the settable setting value in the setting item received by the first receiving unit to the determining unit, the determining unit being configured to determine the setting value for performing the processing based on an instruction from the user; A second receiving unit is configured to receive a setting value from the determining unit; and A fourth transmitting unit is configured to transmit the setting value received by the second receiving unit to the electronic device, and The electronic device performs the processing based on the setting value received by the third receiving unit.

13. A control method for an electronic device, wherein the electronic device performs processing according to a set value, the method comprising: A first sending step is used to send arbitrary text information received as a user input instruction and information relating to at least one setting item of the electronic device to a selection unit, the selection unit being configured to select at least one setting item of the electronic device based on the arbitrary text information. The first receiving step is used to receive setting items from the selection unit; The second sending step is used to send information related to the settable setting value in the setting item received in the first receiving step to the determining unit, the determining unit being configured to determine the setting value for performing the processing according to an instruction from the user; The second receiving step is used to receive the setting value from the determining unit; as well as A control step is used to control the execution of the processing based on the setting value received in the second receiving step.

14. A control method for an electronic device, wherein the electronic device performs processing according to a set value, the method comprising: The third sending step is used to send information related to the settings of the electronic device to the information processing device; as well as The third receiving step is used to receive setting values ​​from the information processing device. The information processing device includes: The fourth receiving unit is configured to receive information relating to settings of the electronic device; A first sending unit is configured to send arbitrary text information received as a user input instruction and information relating to at least one setting item of the electronic device to a selection unit, the selection unit being configured to select at least one setting item of the electronic device based on the arbitrary text information. A first receiving unit is configured to receive settings from the selection unit; The second sending unit is configured to send information related to the settable setting value in the setting item received by the first receiving unit to the determining unit, the determining unit being configured to determine the setting value for performing the processing based on an instruction from the user; A second receiving unit is configured to receive a setting value from the determining unit; and A fourth transmitting unit is configured to transmit the setting value received by the second receiving unit to the electronic device, and The processing is performed based on the setting value received in the third receiving step.

15. A control method for an information processing device, the method comprising: The first sending step is used to send arbitrary text information received as a user input instruction and information relating to at least one setting item of an electronic device that performs processing according to a setting value to a selection unit, the selection unit being configured to select at least one setting item of the electronic device based on the arbitrary text information. The first receiving step is used to receive setting items from the selection unit; The second sending step is used to send information related to the settable setting value in the setting item received in the first receiving step to the determining unit, the determining unit being configured to determine the setting value for performing the processing according to an instruction from the user; The second receiving step is used to receive the setting value from the determining unit; as well as The fourth sending step is used to send the setting value received in the second receiving step to the electronic device.

16. A computer-readable storage medium storing a program for enabling a computer to function as an electronic device according to any one of claims 1 to 9.

17. A computer-readable storage medium storing a program for causing a computer to function as an information processing apparatus according to claim 10.

Images