Information processing method, information processing device, and computer program

Abstract

A desired image information acquisition unit 32 of a camera 12 acquires desired image information relating to an image that a user desires to capture. A prompt generation unit 34 of the camera 12 generates, on the basis of the desired image information, a prompt instructing a generation AI to propose the composition of the image. An answer acquisition unit 36 of the camera 12 acquires composition proposal information for proposing the composition of the image, said composition proposal information being generated by inputting the generated prompt to the generation AI in an LLM server 14.

Description

Information processing method, information processing device, and computer program

[0001] The present disclosure relates to data processing technology, and more particularly to an information processing method, an information processing device, and a computer program.

[0002] In the past, imaging devices have been proposed that assist in setting an imaging composition according to a subject. For example, Patent Literature 1 describes an imaging device that determines a subject type based on a feature amount of a captured image and displays composition assistance information associated with the subject type.

[0003] International Publication No. 2014 / 155789

[0004] The imaging device of Patent Document 1 can only provide composition assistance information associated with subject types pre-registered in a database, and there is a possibility that the proposed composition does not match what the user wants to capture. One objective of the present disclosure is to provide a technology that utilizes generative AI to suggest a composition that matches the image the user wants to capture.

[0005] In order to solve the above problem, an information processing method of one aspect of the present disclosure includes an information processing device that acquires desired image information, which is information regarding an image that a user wishes to capture, and outputs composition suggestion information that suggests a composition for the image, which is generated by inputting input information to the generation AI that instructs the generation AI to suggest a composition for the image based on the desired image information.

[0006] Another aspect of the present disclosure is an information processing device including: a desired image information acquisition unit that acquires desired image information, which is information related to an image that a user wishes to capture; and an output unit that outputs composition suggestion information that suggests a composition of the image, the composition suggestion information being generated by inputting input information that instructs the generation AI to suggest a composition of the image based on the desired image information to the generation AI.

[0007] Any combination of the above components, or any conversion of the expression of the present disclosure between a system, a computer program, a recording medium on which a computer program is recorded, etc., is also valid as an aspect of the present disclosure.

[0008] According to the technology of the present disclosure, generative AI can be utilized to suggest a composition that matches the image the user wishes to capture.

[0009] 1 is a diagram illustrating a configuration of an information processing system according to an embodiment; FIG. 2 is a flowchart illustrating a basic operation of a camera; FIG. 3 is a flowchart illustrating a detailed operation of a camera; FIG. 4 is a diagram illustrating information to be included in a prompt; FIG. 5 is a diagram illustrating an example of a prompt; FIG. 6 is a flowchart illustrating an example of a detailed operation of a camera; and FIG. 7 is a diagram illustrating an example of composition suggestion information. FIGS. 8(a), 8(b), and 8(c) are diagrams illustrating examples of composition suggestion information according to a modified example; and FIGS. 9(a) and 9(b) are diagrams illustrating examples of composition suggestion information.

[0010] The subject of the device or method disclosed herein includes a computer. The computer executes a program to realize the functions of the subject of the device or method disclosed herein. The computer includes, as its main hardware configuration, a processor that operates according to the program. The type of processor is not important as long as it can realize the functions by executing the program. The processor is composed of one or more electronic circuits, including a semiconductor integrated circuit (IC) or a large-scale integration (LSI). The electronic circuits may be integrated into a single chip or may be provided on multiple chips. The multiple chips may be integrated into a single device or may be provided on multiple devices. The program may be recorded on a non-transitory recording medium such as a computer-readable read-only memory (ROM), an optical disk, or a hard disk drive, or on a temporary storage medium such as a computer-readable random access memory (RAM). The program may be stored in advance on a recording medium, or may be supplied to a recording medium or storage medium via a wide area communication network including the Internet.

[0011] An information processing system according to an embodiment utilizes a generative AI to propose a composition suitable for an image a user desires to capture. The generative AI, also known as generative artificial intelligence, is an artificial intelligence system that generates text, images, etc. in response to a prompt. The prompt is input information for the generative AI. Typically, the prompt includes text in natural language describing a task to be performed by the generative AI, in other words, an input sentence to the generative AI. The generative AI used in the system according to an embodiment uses a large-scale language model (hereinafter also referred to as "LLM" (Large Language Models)) that mimics human conversation through text-based dialogue, in other words, an artificial intelligence chatbot. Hereinafter, the term "generative AI" simply refers to an LLM. The generative AI according to an embodiment outputs an answer according to the input prompt.

[0012] 1 shows the configuration of an information processing system 10 according to an embodiment. The information processing system 10 includes a camera 12 and an LLM server 14. The camera 12 and the LLM server 14 are connected via a communication network 16, which may include a LAN, a WAN, the Internet, or the like.

[0013] The camera 12 is an imaging device operated by a user. The camera 12 of the embodiment includes a computer and functions as an information processing device. Images captured by the camera 12 may include both still images and videos. The camera 12 may be a camera mounted on a smartphone or tablet terminal, or may be a digital camera. The LLM server 14 is an information processing device that stores data of the generated AI and provides an interactive service using the generated AI.

[0014] FIG. 1 includes a block diagram showing functional blocks included in the camera 12. The various functions shown in FIG. 1 are realized by a computer installed in the camera 12 executing a program. The computer includes hardware such as a memory into which the program is loaded, one or more processors (e.g., a CPU) that execute the loaded program, an auxiliary storage device, and other LSIs. The processor is composed of multiple electronic circuits including semiconductor integrated circuits and LSIs, and the multiple electronic circuits may be mounted on a single chip or multiple chips. The functional blocks shown in FIG. 1 are realized by cooperation between hardware and software. Therefore, it will be understood by those skilled in the art that these functional blocks can be realized in various ways using hardware alone, software alone, or a combination thereof.

[0015] The camera 12 includes an operation reception unit 20, an imaging unit 22, a storage unit 24, a display unit 26, a communication unit 28, and an imaging support unit 30. The operation reception unit 20 receives operations input by a user via operation members (not shown) such as buttons or a touch panel. The display unit 26 displays various types of information. The display unit 26 may include a liquid crystal display or an organic EL display.

[0016] The imaging unit 22 captures an image of an object (in other words, a subject) within the angle of view in accordance with a predetermined imaging operation input by the user to instruct imaging. The imaging operation can also be considered as an operation to instruct the imaging unit 22 to capture and save a photo.

[0017] The storage unit 24 stores various data. For example, the storage unit 24 stores images captured by the imaging unit 22 in accordance with a user's imaging operation (hereinafter also referred to as "user-captured images"). The communication unit 28 communicates with external devices in accordance with a predetermined communication protocol. The imaging support unit 30 transmits and receives data to and from the LLM server 14 via the communication unit 28.

[0018] The imaging support unit 30, as an information processing device, executes information processing to support the user in imaging. The imaging support unit 30 includes a desired image information acquisition unit 32, a prompt generation unit 34, a response acquisition unit 36, and a device control unit 40. The functions of these multiple functional blocks may be implemented in a computer program, and this computer program may be installed in the storage of the camera 12. The processor of the computer mounted on the camera 12 may perform the functions of these multiple functional blocks by reading this computer program into main memory and executing it.

[0019] The desired image information acquisition unit 32 acquires desired image information, which is information relating to an image that the user wishes to capture. The desired image information acquisition unit 32 may accept desired image information input by the user via an operation member (not shown), such as a button or a touch panel. The desired image information acquisition unit 32 may also acquire desired image information indicated by voice input by the user via a microphone (not shown).

[0020] The desired image information may include the subject of the image the user wishes to capture, or may include both the subject and subtitle of the image the user wishes to capture. The desired image information may also include information indicating the imaging situation input by the user (hereinafter also referred to as "imaging situation information"). The imaging situation information may include information indicating the attitude of the user or the camera 12 acquired by a six-axis inertial sensor or the like, or information indicating the direction in which the user or the camera 12 is facing acquired by a geomagnetic sensor or the like. The desired image information may also include position information related to imaging as imaging situation information. The position information related to imaging may be information indicating the current location of the user or the camera 12.

[0021] The desired image information may also include a user-captured image, which is an image captured in response to an imaging operation input by the user. The desired image information may also include an image captured without a user-capture operation (hereinafter also referred to as a "preview image"). The preview image may be an image captured by the imaging unit 22 and displayed on the display unit 26 while the user has not yet input an imaging operation. The preview image may also be an image depicting the space within the current angle of view of the camera 12. The desired image information may include, as information related to the preview image, spatial information within the current angle of view of the camera 12, such as information about an object present within the angle of view acquired by a time-of-flight (TOF) sensor or the like.

[0022] The prompt generation unit 34 generates a prompt that instructs the generation AI to propose an image composition that conforms to the desired image information, based on the desired image information acquired by the desired image information acquisition unit 32. As described above, when the desired image information includes multiple items, the prompt generation unit 34 may set the multiple items in the prompt.

[0023] The answer acquisition unit 36 ​​inputs the prompt generated by the prompt generation unit 34 to the generation AI by transmitting the prompt to the LLM server 14. The answer acquisition unit 36 ​​acquires answer information generated by the generation AI in response to the prompt, which is transmitted from the LLM server 14. The answer information in the embodiment includes composition suggestion information that proposes a composition of an image in accordance with desired image information.

[0024] The prompt of the embodiment includes information on one or more items specified by the user to be included in the composition suggestion information from the generating AI. In other words, the composition suggestion information includes one or more items specified in the prompt. Below, examples of items included in the composition suggestion information are given. The example items can also be said to be items specified in the prompt.

[0025] The composition suggestion information may include information indicating the impression created by the proposed composition. The impression created by the proposed composition can also be said to be the impression that a person who sees an image to which that composition is applied will have. The composition suggestion information may also include information indicating the reason for proposing each composition. The composition suggestion information may also include instructions to the user for realizing the proposed composition. The instructions to the user may include at least one of information indicating the posture of the user or the camera 12, information indicating an action to be taken by the user, and information indicating the position of the image capture.

[0026] Information indicating the impression brought about by the proposed composition (hereinafter simply referred to as "information indicating the impression") may be stored in the storage unit 24 or a predetermined storage device (such as a server) connected via a network, rather than being included in the composition proposal information and output. This makes it possible to generate other information, such as text or music, based on the information indicating the impression. For example, if text (such as an explanatory text or a catchy slogan) is generated based on the information indicating the impression, the text can be used as an attachment to an image (photograph) captured based on the composition proposal information, enabling secondary use. Furthermore, if music is generated based on the information indicating the impression, the music can be used as music for a slideshow of multiple photos captured based on the composition proposal information, enabling secondary use.

[0027] The composition suggestion information may also include an image (which may be referred to as an image or a forecast image) showing an example of a proposed composition. The composition suggestion information may also include recommended setting values ​​for setting items of a device (the camera 12 in this embodiment) operated by the user. The setting items may also be referred to as imaging parameters, and may include, for example, an aperture (F-number) and a shutter speed.

[0028] The device control unit 40 controls the device based on the composition suggestion information acquired by the response acquisition unit 36. The controlled object may be at least one of the imaging unit 22 and the display control unit 42. The device control unit 40 of the embodiment includes a display control unit 42 and a device setting unit 44. The display control unit 42 displays the composition suggestion information on the display unit 26. The device setting unit 44 sets recommended setting values ​​included in the composition suggestion information for setting items of the camera 12.

[0029] As a modified example, the device control unit 40 may include an audio output control unit. The audio output control unit may control a speaker (not shown) of the camera 12 to output audio indicating the composition suggestion information from the speaker. Furthermore, the device control unit 40 may control the operation or state of imaging support equipment (e.g., a light, a crane, a gimbal, a selfie stick, etc.) provided outside the camera 12 based on the composition suggestion information. Furthermore, the device control unit 40 may control the imaging position in the virtual space and settings of the virtual camera placed in the virtual space (e.g., aperture (F-number), shutter speed, etc.) based on the composition suggestion information.

[0030] The operation of the camera 12 configured as described above will now be described. Fig. 2 is a flowchart showing the basic operation of the camera 12. The desired image information acquisition unit 32 acquires desired image information input by the user (S10). The prompt generation unit 34 generates a prompt including content instructing the user to propose a composition for the image to be captured, based on the desired image information acquired in S10 (S11).

[0031] The answer acquisition unit 36 ​​transmits the prompt generated in S11 to the LLM server 14 (S12), and acquires the composition suggestion information generated by the generation AI in response to the prompt from the LLM server 14 (S13). The device control unit 40 transmits control information for controlling the display content and setting items of the device based on the composition suggestion information acquired in S13 (S14).

[0032] FIG. 3 is a flowchart showing the detailed operation of the camera 12. S20 to S24 in FIG. 3 correspond to S10 in FIG. 2. The desired image information acquisition unit 32 acquires desired image information. The prompt generation unit 34 creates information to be included in the prompt based on the acquired desired image information. FIG. 4 shows the information to be included in the prompt. The information in [ ] in FIG. 4 is information input by the user. Furthermore, the information in < > in FIG. 4 is information obtained based on input from the user.

[0033] Specifically, the user may input the subject and subtitle of the image they wish to capture as the desired image information. In this case, as shown in FIG. 4 , the desired image information acquisition unit 32 may accept input information specifying subject AAA and subtitle BBB (S20). The prompt generation unit 34 may generate a sentence such as "The subject of the photo I want to take is AAA, and the subtitle is BBB" as information to be included in the prompt. Note that the user may directly input a sentence, word, or keyword indicating the subject and subtitle of the desired image information, or may input the subject and subtitle by selecting an option.

[0034] 4, the desired image information acquisition unit 32 may receive input information indicating Who (imaging target), Where (imaging location), When (imaging date and time), and What (action / status) of the 5W1H as the imaging situation information (S21). The prompt generation unit 34 may generate information indicating Who (imaging target), Where (imaging location), When (imaging date and time), and What (action / status) as information to be included in the prompt.

[0035] The imaging situation information may include information about an event occurring at the object (or imaging location) being imaged. The event information may be static information or dynamic information. Static information may be, for example, (1) information about the lighting time of a tower or area, or (2) information about a train timetable. The information (1) and (2) may be information related to the above 4Ws. For example, the information (1) can be said to be information suggesting whether the lighting (operation / status) of a tower (imaged object) or an area (imaged location) is occurring at the image capture date and time. Furthermore, the information (2) can be said to be information suggesting whether a given train (imaged object) is running or stopped (operation / status) at the image capture location and image capture date and time.

[0036] By including such information in the imaging situation information, for example, the generation AI can consider whether the area is lit up at the time of imaging and, if lit up, can propose a composition that avoids backlighting. Furthermore, the generation AI can consider how many minutes it will take for the train to be imaged at the imaging location and imaging date and time to be imaged (i.e., when it passes or stops), and include instructions to the user, such as how many minutes to wait, in the proposal. In this way, more appropriate composition proposal information can be output. Note that the static information may be input by the user, or may be searched and obtained from a specified database, the Internet, etc., based on the imaging situation information input by the user.

[0037] Furthermore, the dynamic information may be, for example, predicted information about the timing at which a bird will take off or predicted information about the time at which a racing car will pass in front of the bird's eyes. Such information may be information about the above-mentioned 4Ws that is predicted or estimated based on a preview image (e.g., paragraph 0043). Unlike static information, the dynamic information may be information that can be predicted point by point based on a preview image (paragraph 0043). For example, the dynamic information may be information obtained by the generation AI generating predictive information about the number of seconds the bird is likely to take off based on the analysis results when it is identified through image analysis processing based on the preview image that the bird has spread its wings.

[0038] By including such information in the imaging situation information, the generation AI can include in the proposal instructions to the user, for example, instructions to take a picture within a certain number of seconds, taking into account information on the possibility that the bird to be imaged will take off within a certain number of seconds. Similarly, in the case of a racing car, the proposal can include user instructions such as how many seconds to wait based on predicted information on how many seconds it will take for the car to pass by.

[0039] 4, the desired image information acquisition unit 32 may receive input information indicating the "Why" (reason for capturing images) and "How" (imaging method) of the 5W1H as the imaging situation information (S21). The prompt generation unit 34 may generate information indicating the reason or motivation for capturing images and the desired shooting situation (how the user wants to capture the subject) as information to be included in the prompt. Note that the user may directly input a sentence, word, or keyword indicating the imaging situation information as the desired image information, or may input the imaging situation information by selecting an option.

[0040] The user may also input their current location as the desired image information. In this case, as shown in FIG. 4 , the desired image information acquisition unit 32 may accept the input current location information (S22). The desired image information acquisition unit 32 may acquire the current location of the camera 12 using a global positioning system (GPS) or a global navigation satellite system (GNSS). The prompt generation unit 34 may generate information describing the location where the image is being captured as information to be included in the prompt. The current location information and the information to be included in the prompt may include information such as an address, a building name, latitude, and longitude.

[0041] The user may also input a captured user-captured image as the desired image information. In this case, as shown in FIG. 4 (Example 1 of S23), the desired image information acquisition unit 32 may accept input information including the image analysis result of the user-captured image and additional information indicating a desire to take a photo with a better composition (S23). A publicly known technique may be used for the image analysis. The prompt generation unit 34 may generate information to be included in the prompt, including the image analysis result, such as "There is a tree (What) in the upper right corner of the field of view (Where)" or "There are two people (Who) in the center of the field of view (Where)," and the additional information.

[0042] 4 (Example 2 of S23), the desired image information acquisition unit 32 may accept input information including image data of the user-captured image and additional information indicating a desire to take a photo with a better composition (S23). The prompt generation unit 34 may generate information to be included in the prompt, including the image data of the user-captured image, an instruction to analyze the user-captured image, and the additional information.

[0043] The user may also input a preview image as the desired image information. In this case, as shown in FIG. 4, the desired image information acquisition unit 32 may accept input information including the image analysis results of the preview image (S24). The prompt generation unit 34 may generate information including the image analysis results, such as "The scenery currently captured by the camera is LLL," as information to be included in the prompt. As a variant, the information to be included in the prompt may include image data of the preview image. In the embodiment, the desired image information acquisition unit 32 executes at least one process among S20 to S24. The desired image information acquisition unit 32 may execute multiple processes among S20 to S24.

[0044] When a user-captured image and / or a preview image is input, the desired image information acquisition unit 32 or the prompt generation unit 34 may perform image analysis processing to identify the content of the image. Note that the image analysis processing may be performed by an image analysis unit (not shown) rather than by the desired image information acquisition unit 32 or the prompt generation unit 34. In this case, the prompt generation unit 34 may acquire image analysis information from the image analysis unit.

[0045] The prompt generation unit 34 generates a prompt based on the information to be included in the prompt generated in S20 to S24 (S26). For example, the prompt generation unit 34 may include in the prompt a sentence that concatenates the information to be included in the prompt generated in S20 to S24.

[0046] The user may also input information about objects that should not be captured in the captured image as desired image information. In this case, the desired image information acquisition unit 32 receives input information specifying the objects that should not be captured in the captured image in addition to any of S20 to S24. The prompt generation unit 34 includes information about the objects that should not be captured in the captured image in the prompt.

[0047] Information about objects that should not appear in a captured image (herein also referred to as "excluded objects") may be included in prompts or composition suggestion information and output. In addition, when performing image processing on an image (photograph) captured based on the composition suggestion information, the information may be used as information for identifying objects to be processed. For example, if a composition is suggested based on information about the excluded objects, but the excluded objects inevitably appear in the captured image, it is expected that image processing will be performed on the captured image to remove the excluded objects. In such a case, the information processing device (camera 12, etc.) can automatically perform image processing to identify and remove objects to be removed (excluded objects) based on the information about the excluded objects, even if the user does not specify or input the objects to be removed.

[0048] Furthermore, when the user inputs information on items to be included in the composition proposal information (hereinafter also referred to as "proposal items"), the desired image information acquisition unit 32 acquires the input information on the proposal items. The prompt generation unit 34 incorporates the information on the proposal items into the prompt, separately from the "information to be included in the prompt" generated based on the desired image information.

[0049] The proposed items may include (1) the name, (2) description, (3) reason for adoption, (4) the impression given by the composition, (5) the captured image, (6) instructions to the user for realizing the composition, and (7) recommended settings of the equipment for realizing the composition, etc., regarding the composition proposed by the generating AI. The captured image is an image that is expected to be captured if the proposed composition is adopted. Note that if predetermined proposed items (in other words, default proposed items) are determined in advance, the proposed items do not necessarily have to be input by the user. Furthermore, the user may input information to add desired new proposed items to the default proposed items.

[0050] 5 shows an example of a prompt 56. The prompt 56 includes "information to be included in the prompt" based on the desired image information, desired proposal content indicating proposed items, and a question based on that information. The "information to be included in the prompt" based on the desired image information includes the subject, subtitle, and information on objects that should not be included in the image. The prompt 56 specifies the subject "Rouge Rose," the subtitle "Sea and Sky," and the object that should not be included in the image "Car." Additionally, the proposed items include the name of the composition, the reason for the proposal, the captured image, the impression of the composition, and recommended settings for the camera's imaging parameters.

[0051] The prompt may include "content suggesting that the generation AI can perform a web search and obtain the necessary information by itself." This allows the LLM to, for example, obtain information about popular compositions on SNS (Social Networking Service) or information about general compositions corresponding to the input subject and subtitle (e.g., a bird's-eye view composition for cooking) through a web search, and generate composition suggestion information by referring to the obtained information.

[0052] 3, the answer acquisition unit 36 ​​transmits the generated prompt to the LLM server 14 (S27). The answer acquisition unit 36 ​​acquires the answer information generated by the generation AI in response to the prompt from the LLM server 14 (S28).

[0053] The generation AI of the embodiment generates answer information indicating an additional question to ask the user when the prompt does not provide sufficient information for proposing a composition. When information for proposing a composition is insufficient, the prompt generation unit 34 may include content instructing the user to answer an additional question in the prompt. The content instructing the user to answer an additional question may be, for example, "Please consider whether this prompt is sufficient to suggest a composition. If not, please ask an additional question."

[0054] Furthermore, a generated AI that interprets the prompt and a generated AI that has been trained as a professional photographer may be provided separately. In this case, the instruction to ask the user additional questions may be, for example, "Please ask the generated AI that has been trained as a professional photographer whether this prompt is sufficient to suggest a composition. If the answer is that it is not sufficient, please let me think of additional questions regarding the missing information. Then, please ask me those additional questions."

[0055] If the answer information acquired in S28 includes an additional question (Y in S29), the display control unit 42 causes the display unit 26 to display the additional question (S30). The user inputs an answer to the additional question into the camera 12 via the operation means of the camera 12. The answer to the additional question can also be considered new desired image information. The desired image information acquisition unit 32 acquires the answer to the additional question input by the user (S31). Thereafter, the process returns to S26, and the prompt generation unit 34 generates a new prompt including the answer to the additional question, and the answer acquisition unit 36 ​​transmits the new prompt to the LLM server 14. If the answer information acquired in S28 does not include an additional question, that is, if the answer information is composition suggestion information (N in S29), the process proceeds to the process shown in FIG. 5.

[0056] Fig. 6 is also a flowchart showing a detailed example of the operation of the camera 12. Fig. 5 shows the operation following Fig. 3. If the composition suggestion information includes the name of a composition to be proposed (hereinafter also referred to as "proposed composition"), the display control unit 42 causes the display unit 26 to display the name of the proposed composition (S40). If the composition suggestion information includes information on the impression created by the proposed composition, the display control unit 42 causes the display unit 26 to display the information on the impression (S41). If the composition suggestion information includes a reason for proposing the composition, the display control unit 42 causes the display unit 26 to display the reason for the proposal (S42).

[0057] If the generation AI has an image generation function, the generation AI may generate a captured image using specific diagrams or images in addition to a captured image using text information, and may include the captured image using the diagrams or images in the composition suggestion information. For example, the composition suggestion information may suggest multiple types of compositions to the user and may include captured image using text information or images for each of the multiple types of compositions. If the composition suggestion information includes a captured image using the proposed composition (a captured image using a diagram or image), the display control unit 42 displays the captured image on the display unit 26 (S43).

[0058] A modified example of S43 will be described. If the generation AI does not or cannot generate a captured image using a diagram or image, the LLM server 14 may perform a web search using a Retrieval-Augmented Generation (RAG) mechanism and provide composition suggestion information that includes images on the web as reference information. Note that only a predetermined number of images (e.g., one) of images on the web may be included in the composition suggestion information, or a link (such as a URL) to the image on the web may be included in the composition suggestion information. Note that the search target or link destination is not limited to the web, and may also be a predetermined server or database.

[0059] Furthermore, the prompt generation unit 34 of the camera 12 may use the RAG mechanism described above to acquire an image as a user-captured image from the web or the like and set it as a prompt. Specifically, the prompt generation unit 34 may receive information input by the user about a desired user-captured image to be set as a prompt (e.g., an image of a recently opened spot, etc.), and may search the Internet to acquire an image that matches the user-captured image. The image capture support unit 30 may analyze the acquired image and set the analysis results (e.g., image feature information) as the prompt. The input may be input items (e.g., subject and subtitle) such as those in S20 to S22 of FIG. 3 of the embodiment. In this case, for example, images may be searched for on the web or the like based on the subject and subtitle, and the hit image may be acquired as the user-captured image.

[0060] Returning to the description of Fig. 6 , if the composition suggestion information includes instructions to the user for realizing the proposed composition, the display control unit 42 causes the display unit 26 to display the instructions (S44). The instructions to the user may include sentences such as "point the camera up" or "bend over." If the composition suggestion information includes recommended settings of the device for realizing the proposed composition, the display control unit 42 causes the display unit 26 to display the recommended settings (S45).

[0061] In this embodiment, the process corresponding to the proposed item is usually performed among S40 to S45. If an item that is not included in the proposed items is included in the composition proposal information, the display control unit 42 may or may not display that item. Furthermore, if an item that is included in the proposed items is not included in the composition proposal information, the display control unit 42 may display a message indicating that information about that item is not included.

[0062] FIG. 7 shows an example of composition suggestion information 50. The composition suggestion information 50 in FIG. 7 suggests a diagonal composition based on the information (subject, subtitle, etc.) specified in the prompt 56 in FIG. 5. The composition suggestion information 50 in FIG. 7 includes the proposal items specified in the prompt 56 in FIG. 5, namely, the composition name, the reason for the suggestion, the captured image, the impression of the composition, and recommended settings for the camera's imaging parameters. The imaging parameters include, for example, the aperture (F-number), shutter speed, ISO sensitivity, and focus target. As described above, the recommended settings for the imaging parameters are presented to the user (S45 in FIG. 5), and if automatic setting is enabled, the recommended settings are automatically set for the imaging parameters of the camera 12.

[0063] 8(a), 8(b), and 8(c) show examples of composition suggestion information 50 according to modified examples. The composition suggestion information 50 in each of Figs. 8(a), 8(b), and 8(c) includes the name of the proposed composition, the reason for the proposal, and an impression.

[0064] In the embodiment, the user specifies the subject and subtitle of the image to be captured. However, as a variant, the user may specify the subject but not the subtitle. As another variant, the user may specify the imaging situation information, the user-captured image, the preview image, etc., but not the subject and subtitle. In this case, the prompt may include content instructing the system to estimate the subject and subtitle based on the imaging situation information, the user-captured image, the preview image, etc., and to propose an appropriate composition based on the estimation results. For example, FIG. 8( a) shows an example in which the user does not specify the subject and subtitle, and the generation AI estimates the subject as AAA and the subtitle as BBB. FIG. 8( b) shows an example in which the user specifies the subject as CCC but does not specify the subtitle. FIG. 8( c) shows an example in which the user specifies the subject as DDD and the subtitle as EEE.

[0065] 6 , the display control unit 42 causes the display unit 26 to display content inquiring of the user as to whether or not the user is satisfied with the composition suggestion information (i.e., the composition suggested by the generation AI). The imaging support unit 30 acquires information input by the user as to whether or not the user is satisfied with the composition suggestion information.

[0066] If the user inputs that he or she is dissatisfied with the composition proposal information (N in S46), the prompt generation unit 34 generates a new prompt (S49). The new prompt includes content instructing the generation AI to respond with information necessary to propose a composition different from the composition proposed in the previous composition proposal information. For example, the new prompt may include the sentence, "I would like you to propose a composition different from the compositions you have proposed so far. Please tell me the information you need to do so."

[0067] The answer acquisition unit 36 ​​transmits the new prompt generated in S49 to the LLM server 14, and acquires answer information indicating necessary additional information generated by the generation AI in response to the new prompt from the LLM server 14 (S50). The display control unit 42 displays the necessary additional information indicated by the answer information on the display unit 26 (S51). For example, if only a subject and a subtitle were specified in the previous prompt, the necessary additional information may be at least one of imaging situation information, a user-captured image, and a preview image.

[0068] The prompt generation unit 34 acquires the additional information input by the user to the camera 12 (S52). After this, the process returns to S26 in Fig. 3, where the prompt generation unit 34 generates a new prompt including the additional information, and the answer acquisition unit 36 ​​transmits the new prompt to the LLM server 14. In this way, by acquiring from the generation AI the information necessary to have the generation AI propose a new composition and presenting it to the user, the user can be assisted in receiving suggestions for new compositions.

[0069] If the user indicates satisfaction with the composition proposal information (Y in S46), the device control unit 40 checks whether the composition proposal information includes recommended settings for the camera 12's imaging parameters and whether automatic setting of the recommended settings is enabled in the camera 12. If the composition proposal information does not include the recommended settings for the camera 12 or automatic setting of the recommended settings is disabled (N in S47), the processing in this figure ends. If the composition proposal information includes the recommended settings for the camera 12 and automatic setting of the recommended settings has been enabled in advance (Y in S47), the device setting unit 44 automatically sets the recommended settings for the imaging parameters of the camera 12 (S48). This reduces the burden on the user who attempts to capture an image according to the proposed composition.

[0070] 9(a) and 9(b) show display examples of composition suggestion information 50. Figures 9(a) and 9(b) show instruction information 54 for the user that is included in the composition suggestion information 50. The instruction information 54 here includes instructions for the user to realize the suggested composition, and can also be considered guide information that supports the user in capturing an image in accordance with the suggested composition.

[0071] The instruction information 54 in FIG. 9A is information indicating the direction in which the user should move, and is placed on a preview image 52, which is an image being captured by the camera 12. The display control unit 42 displays the instruction information 54 indicating the direction superimposed on the preview image 52. The instruction information 54 may include a text message, such as "move left," indicating the direction in which the user should move to realize the proposed composition. Furthermore, as shown in FIG. 9A, the instruction information 54 may include an image, such as an arrow, indicating the direction in which the user should move to realize the proposed composition. Note that the instruction information 54 may also be information indicating the direction in which the camera 12 should be pointed. In this case, the instruction information 54 may include a text message, such as "point the camera left," indicating the direction in which the camera 12 should be pointed to realize the proposed composition. Furthermore, the instruction information 54 may include an image, such as an arrow, indicating the direction in which the camera 12 should be pointed to realize the proposed composition.

[0072] The imaging support unit 30 of the camera 12 may acquire the direction in which the user is facing based on the detection results of a geomagnetic sensor or the like (not shown). The imaging support unit 30 may also acquire the user's current location (which can also be considered the current location of the camera 12) acquired by a GPS or the like (not shown). The imaging support unit 30 also acquires a preview image, which is the video being captured by the camera 12, in real time.

[0073] The imaging support unit 30 may calculate the instruction information 54 based on the direction in which the user is facing, information about the user's current location, and a captured image (here, a captured image using a diagram or image) provided by the LLM server 14 (generation AI). For example, the imaging support unit 30 may acquire an image of the area around the location corresponding to the current location from a virtual globe system on the Internet, and compare the image of the area around the location corresponding to the current location with the captured image to calculate the position and direction in which the user should face in order to capture the image shown in the captured image, and generate the instruction information 54 based on the calculation result.

[0074] 9B is an image (e.g., a captured image) of the image capture result when capturing an image using the proposed composition, which is placed on the video (preview image 52) being captured by the camera 12. The display control unit 42 may display the image (shown by a dashed line) as the instruction information 54 by superimposing it on the preview image 52. The display control unit 42 may display the image with a predetermined transparency set.

[0075] For example, the imaging support unit 30 may acquire an image of the vicinity of the location corresponding to the current location from a virtual globe system on the Internet, and identify a position where the captured image can be captured (in other words, a position where an image can be captured with the proposed composition) by comparing the image of the vicinity of the location corresponding to the current location with the captured image provided by the LLM server 14. The imaging support unit 30 may identify a position that falls within the angle of view of the preview image based on the direction in which the user is facing and information about the user's current location, and when the position where the captured image can be captured falls within the angle of view of the preview image, may superimpose an image as instruction information 54 on the preview image 52.

[0076] The instruction information 54 can clearly show the user how to move to capture an image with the proposed composition (for example, by moving so that the tree is in the center of the angle of view). The user changes the orientation or moves the camera 12 according to the instruction information 54, and determines whether the captured image matches the composition proposed by the generation AI. The user performs an image capture operation when the captured image matches the composition proposed by the generation AI. Note that the imaging unit 22 of the camera 12 may detect when the captured image matches the composition proposed by the generation AI, and automatically capture and save a photo.

[0077] According to the information processing system 10 of the embodiment, by utilizing the generation AI, it is possible to suggest to the user a composition that is in line with the content that the user wants to capture, thereby supporting the user in capturing images. For example, according to the information processing system 10 of the embodiment, it is possible to suggest a composition that is in line with the subject and subtitle of the image that the user wants to capture.

[0078] Furthermore, the camera 12 of the embodiment can support the user in capturing images by controlling the device based on the composition suggestion information generated by the generation AI. Furthermore, the information processing system 10 of the embodiment can further effectively support the user in capturing images by including items desired by the user in the composition suggestion information.

[0079] The present disclosure has been described above based on the embodiments. The embodiments are merely examples, and it will be understood by those skilled in the art that various modifications are possible in the combination of each component or each treatment process of the embodiments, and that such modifications are also within the scope of the present disclosure.

[0080] A first modified example will be described. In the embodiment, the camera 12 executes a process to assist the user in capturing images using the camera 12. As a modified example, an information processing device in a separate housing from the camera 12 may execute a process to assist the user in capturing images using the camera 12. In this case, the information processing device may include a communication unit, a processing unit, and a storage unit. The communication unit may communicate with the camera 12 and the LLM server 14 via the communication network 16. The storage unit may correspond to the storage unit 24. The processing unit may have the functions of the image capturing support unit 30. Furthermore, the information processing device may store data of the generation AI. In this case, the information processing device may execute the image capturing support process and the content generation process by the generation AI.

[0081] The information processing device may utilize a generation AI to assist in capturing images in a virtual space. In this case, the prompt generation unit 34 of the information processing device may include position information (such as coordinate information) of various objects placed in the virtual space in the prompt. This allows the generation AI to suggest a composition for capturing an image of the virtual space with a virtual camera based on the positions of each object placed in the virtual space.

[0082] A second modified example will be described. In the embodiment, the LLM server 14 provided a chat service using the generated AI. As a modified example, the camera 12 may store data of the generated AI. In this case, the camera 12 may perform imaging processing, imaging support processing by the imaging support unit 30, and content generation processing by the generated AI.

[0083] Any combination of the above-described embodiments and multiple modifications is also useful as an embodiment of the present disclosure. A new embodiment resulting from a combination combines the effects of the combined embodiments and modifications. It will also be understood by those skilled in the art that the functions to be performed by each component recited in the claims can be realized by each component shown in the embodiments and modifications, either alone or in combination.

[0084] <Supplementary Notes> The above embodiments and variations disclose the following technologies. [Technology 1-1] An information processing method in which an information processing device acquires desired image information, which is information about an image a user wishes to capture, and outputs composition suggestion information that suggests a composition for the image, the composition being generated by inputting input information to a generation AI that instructs the generation AI to suggest a composition for the image based on the desired image information. This information processing method can provide information generated by the generation AI that suggests a composition that is consistent with the content the user wishes to capture, thereby supporting the user's capture. [Technology 1-2] The information processing method described in Technology 1-1, in which the desired image information includes a subject of the image. This information processing method can provide information that suggests a composition that is consistent with the subject of the image to be captured, thereby supporting the user's capture. [Technology 1-3] The information processing method described in Technology 1-2, in which the desired image information further includes a subtitle of the image. This information processing method can provide information that suggests a composition that is consistent with the subject and subtitle of the image to be captured, thereby supporting the user's capture. [Technology 1-4] The information processing method according to Technology 1-1, wherein the desired image information includes information indicating the imaging situation. This information processing method can provide information that suggests a composition that is appropriate for the imaging situation, thereby supporting the user's imaging. [Technology 1-5] The information processing method according to Technology 1-1, wherein the desired image information is generated based on information input by the user regarding an image the user wishes to capture. This information processing method can provide information that suggests a composition that is appropriate for the image the user wishes to capture, thereby supporting the user's imaging. [Technology 1-6] The information processing method according to Technology 1-1, wherein the information processing device accepts input of an arbitrary captured image related to the image, and when the arbitrary captured image is input, the desired image information is generated based on the captured image. This information processing method can provide information that suggests a composition that is appropriate for an existing captured image, thereby supporting the user's imaging. [Technology 1-7] The information processing method according to Technology 1-6, wherein the captured image is an image captured in response to an imaging operation by the user.This information processing method can provide information that suggests a composition that is in line with the content of images the user has taken up to that point, thereby assisting the user in taking pictures. [Technology 1-8] The information processing method described in Technology 1-7, in which the captured image includes a preview image acquired through a camera. This information processing method can provide information that suggests a composition that is in line with the user's surrounding environment, thereby assisting the user in taking pictures. [Technology 1-9] The information processing method described in Technology 1-1, in which the desired image information includes location information related to the image capture. This information processing method can provide information that suggests a composition that is in line with the location where the image capture will be performed, thereby assisting the user in taking pictures. [Technology 1-10] The information processing method described in Technology 1-6, in which the desired image information is generated by estimating the subject contained in the captured image through image analysis from the captured image. This information processing method can provide information that suggests a composition that is in line with the subject contained in the captured image, thereby assisting the user in taking pictures. [Technology 1-11] The information processing method according to Technology 1-6, wherein at least one of the following information is estimated from the captured image through image analysis: a subject of the captured image, information indicating the capture conditions of the captured image, and location information related to the capture of the captured image; and the desired image information is generated based on the estimated information. This information processing method can provide information suggesting a composition based on the subject of the captured image, the capture conditions, or the capture location information, thereby assisting the user in capturing images. [Technology 1-12] The information processing method according to Technology 1-1, wherein the desired image information includes information about objects that should not be captured in the image. This information processing method can provide information suggesting a composition that does not include specific objects, thereby assisting the user in capturing images. [Technology 1-13] The information processing method according to Technology 1-1, wherein input regarding suggested items to be included in the composition suggested information is accepted, and input information based on the suggested items is input to the generation AI. This information processing method can provide composition suggested information including specified suggested items, thereby assisting the user in capturing images.[Technology 1-14] An information processing device comprising: a desired image information acquisition unit that acquires desired image information, which is information about an image that a user wishes to capture; and an output unit that outputs composition proposal information that proposes a composition for the image, generated by inputting input information to the generation AI that instructs the generation AI to propose a composition for the image based on the desired image information. This information processing device can provide information that proposes a composition that is consistent with the content that the user wants to capture, generated by the generation AI, thereby supporting the user's capture. [Technology 1-15] A computer program that causes an information processing device to execute the following steps: acquire desired image information, which is information about an image that a user wishes to capture; and output composition proposal information that proposes a composition for the image, generated by inputting input information to the generation AI that instructs the generation AI to propose a composition for the image based on the desired image information. This computer program can provide information that proposes a composition that is consistent with the content that the user wants to capture, thereby supporting the user's capture.

[0085] [Technology 2-1] An information processing method in which an information processing device controls an imaging device based on composition proposal information that suggests a composition for an image to be captured by a user, the composition proposal information being generated by inputting input information to a generation AI that instructs the generation AI to suggest a composition for the image to be captured by the user. This information processing method can support a user's imaging by controlling the imaging device based on the composition proposal information generated by the generation AI. [Technology 2-2] The information processing method described in Technology 2-1, in which the input information includes information on items to be included in the composition proposal information, specified by the user. This information processing method can include items desired by the user in the composition proposal information, thereby more effectively supporting the user's imaging. [Technology 2-3] The information processing method described in Technology 2-1, in which the imaging device includes a display, and controlling the imaging device includes controlling the composition proposal information to output the composition proposal information to the display. This information processing method can visually present the composition proposal information to the user. [Technology 2-4] The information processing method described in Technology 2-3, in which the composition proposal information includes information indicating the impression created by the proposed composition. According to this information processing method, by presenting the user with the impression created by the proposed composition, it is possible to assist the user in deciding whether or not to adopt the proposed composition. [Technology 2-5] The information processing method according to Technology 2-3, wherein the composition proposal information includes information indicating the reason for proposing the composition. According to this information processing method, by presenting the user with the reason for proposing the composition, it is possible to assist the user in deciding whether or not to adopt the proposed composition. [Technology 2-6] The information processing method according to Technology 2-3, wherein the composition proposal information includes instructions for the user to realize the proposed composition. According to this information processing method, it is possible to assist the user in realizing the proposed composition. [Technology 2-7] The information processing method according to Technology 2-6, wherein the composition proposal information includes instructions for the user to realize the proposed composition, generated based on the user's current location and the proposed composition. According to this information processing method, it is possible to assist the user in realizing the composition proposed by the generation AI.[Technology 2-8] The information processing method according to Technology 2-7, wherein the instructions to the user include information indicating the direction in which the user should move to achieve the proposed composition. This information processing method makes it possible to clearly present the direction in which the user should move to achieve the composition proposed by the generation AI. [Technology 2-9] The information processing method according to Technology 2-7, wherein the instructions to the user include an image of the image result when captured with the proposed composition, displayed together with a preview image acquired through a camera. This information processing method makes it possible to clearly present to the user how to move to achieve the composition proposed by the generation AI. [Technology 2-10] The information processing method according to Technology 2-3, wherein the composition proposal information includes an image showing an example of the proposed composition. This information processing method makes it possible to support the user in deciding whether or not to adopt the proposed composition by presenting an image showing an example of the composition to the user. [Technology 2-11] The information processing method according to Technology 2-1, wherein the composition proposal information includes recommended setting values ​​for imaging parameters suitable for the composition, and controlling the imaging device includes controlling the imaging parameters of the imaging device based on the recommended setting values. This information processing method can reduce the burden on the user of capturing an image with a composition suggested by the composition proposal information. [Technology 2-12] The information processing device: acquires information on whether the user is satisfied with the composition proposal information; and if the user is dissatisfied with the composition proposal information, generates new input information that instructs the generation AI to respond with information necessary to propose a composition different from the composition suggested in the composition proposal information; and acquires and outputs a response indicating the information necessary to propose the different composition, generated by inputting the new input information into the generation AI. This information processing method can support a user who wishes to receive a new composition proposal by presenting the user with information necessary to have the generation AI propose a new composition.[Technology 2-13] An information processing device including a device control unit that controls an imaging device based on composition proposal information that suggests a composition for an image to be captured, which is generated by inputting input information to the generation AI that instructs the generation AI to suggest a composition for the image to be captured by the user. With this information processing device, it is possible to assist a user in capturing an image by controlling the imaging device based on the composition proposal information generated by the generation AI. [Technology 2-14] A computer program that causes an information processing device to execute the following: controlling the imaging device based on composition proposal information that suggests a composition for the image, which is generated by inputting input information to the generation AI that instructs the generation AI to suggest a composition for the image to be captured by the user. With this computer program, it is possible to assist a user in capturing an image by controlling the imaging device based on the composition proposal information generated by the generation AI.

[0086] The technology of the present disclosure is applicable to information processing devices and information processing apparatuses.

[0087] REFERENCE SIGNS LIST 10 Information processing system, 12 Camera, 14 LLM server, 30 Imaging support unit, 32 Desired image information acquisition unit, 34 Prompt generation unit, 36 Answer acquisition unit, 40 Device control unit, 42 Display control unit, 44 Device setting unit.

Claims

1. An information processing method in which an information processing device acquires desired image information, which is information about an image a user wishes to capture, generates composition suggestion information that suggests a composition for capturing the image based on the desired image information, and outputs the generated composition suggestion information.

2. The information processing method of claim 1, wherein the composition suggestion information is generated by inputting input information based on the desired image information into a generation AI, and the input information includes instructions for causing the generation AI to suggest the composition.

3. The information processing method according to claim 2, wherein the desired image information includes at least one of the following information: the subject of the image, information indicating the circumstances of the image capture, and location information related to the image capture.

4. The information processing method according to claim 2, wherein the information processing device accepts input of information relating to an image the user wishes to capture, and the desired image information is generated based on the input information relating to the image the user wishes to capture.

5. The information processing method according to claim 4, wherein, when an arbitrary captured image related to the image is input as information regarding the image the user desires to capture, the desired image information is generated based on the captured image.

6. The information processing method according to claim 5, wherein the desired image information is generated based on information about a subject included in the captured image, estimated from the captured image through image analysis.

7. The information processing method according to claim 6, wherein the desired image information is generated based on at least one of the following information estimated through image analysis of the captured image: the subject of the captured image, information indicating the circumstances under which the captured image was captured, and location information related to the capture of the captured image.

8. The information processing method according to claim 5, wherein the captured image is an image captured in response to an imaging operation in an imaging device, or a preview image acquired through an imaging operation.

9. The information processing method according to claim 2, wherein the desired image information includes information about objects that should not be captured in the image.

10. The information processing method of claim 2, wherein the information processing device accepts input regarding proposed items to be included in the composition proposal information, and the input information includes instructions for causing the generation AI to generate the composition proposal information including the proposed items.

11. The information processing method according to claim 1, wherein the information processing device further controls an imaging device based on the output composition proposal information.

12. The information processing method according to claim 11, wherein the imaging device includes a display, and the control of the imaging device includes control of outputting the composition suggestion information to the display.

13. The information processing method according to claim 12, wherein the composition suggestion information includes at least one of the following information: information regarding the impression created by the proposed composition; information indicating the reason for suggesting the composition; and an image showing an example of the proposed composition.

14. The information processing method according to claim 12, wherein the composition suggestion information includes instructions to the user for realizing the proposed composition, generated based on the user's current location and the proposed composition.

15. The information processing method according to claim 14, wherein the instructions to the user include information indicating a direction in which the user should move or a direction in which the user should face in order to realize the proposed composition.

16. The information processing method according to claim 12, wherein the composition suggestion information includes an image of the image result when an image is captured using the proposed composition, and is output to the display together with a preview image acquired through an imaging operation in the imaging device.

17. The information processing method according to claim 11, wherein the composition suggestion information includes recommended setting values ​​of imaging parameters suitable for the composition, and the control of the imaging device includes control of setting the imaging parameters of the imaging device based on the recommended setting values.

18. The information processing method of claim 2, wherein, after outputting the composition proposal information, the information processing device generates new input information that instructs the generation AI to respond with information necessary to propose a composition different from the composition proposed in the composition proposal information, and outputs a response indicating the information necessary to propose the different composition, generated by inputting the new input information to the generation AI.

19. An information processing device comprising: a desired image information acquisition unit that acquires desired image information, which is information regarding an image that a user wishes to capture; a generation unit that generates composition proposal information that proposes a composition for capturing the image based on the desired image information; and an output unit that outputs the generated composition proposal information.

20. A computer program for causing an information processing device to perform the following steps: acquire desired image information, which is information regarding an image that a user wishes to capture; generate composition proposal information that proposes a composition for capturing the image based on the desired image information; and output the generated composition proposal information.

Images