Image processing device, operating method for image processing device, and operating program

Abstract

An image processing device includes a processor. The processor acquires capture environment information of periods before and after the main capture timing of images including at least one of a still image or a moving image, and executes a related process related to a main image which is the main captured image on the basis of the capture environment information.

Description

Image processing device, operation method and operation program for image processing device

[0001] The technology of the present disclosure relates to an image processing device, an operating method for the image processing device, and an operating program.

[0002] Japanese Patent Application Laid-Open Publication No. 2015-233188 describes an information processing device that includes a processing unit that extracts one or more frames from a moving image based on specific instruction information that is set in accordance with input audio, and records frame identification information that identifies the extracted frames in a storage unit.

[0003] Japanese Patent Application Laid-Open Publication No. 2022-110177 describes a golf digest creation system having a data analysis unit that performs image analysis of video data and / or audio analysis of audio data, and a digest creation unit that extracts highlight scenes based on the results of the data analysis unit.

[0004] One embodiment of the technique of the present disclosure provides an image processing device, an operating method for an image processing device, and an operating program that can perform related processing related to a real image captured more appropriately than ever before.

[0005] The image processing device according to the disclosed technology includes a processor, which acquires shooting environment information for a period before and after the timing of the actual shooting of an image including at least one of a still image or a video, and performs related processing related to the actual image, which is the image that was actually shot, based on the shooting environment information.

[0006] The filming environment information may include associated audio for the period.

[0007] The shooting environment information may include related images for a period of time.

[0008] The related process may be a process of extracting at least a portion of the original image.

[0009] The related processing may be an additional information adding processing for adding additional information to be added to the main image based on the shooting environment information.

[0010] The related processing may be processing for generating, based on the image-capturing environment information, a related image that is related to the main image during the period in which the image-capturing environment information is acquired.

[0011] The processor may store the shooting environment information in a storage format with a reduced amount of data based on the analysis result of the content of the shooting environment information.

[0012] The shooting environment information may be acquired by a device other than the camera that captures the main image.

[0013] The shooting environment information may include at least one of a user's movement history related to the actual shooting and a user's search history related to the actual shooting.

[0014] The method of operating an image processing device according to the disclosed technology is a method of operating an image processing device equipped with a processor, in which the processor acquires shooting environment information for periods before and after the timing of the actual shooting of an image including at least one of a still image or a video, and performs related processing related to the actual image, which is the image that was actually shot, based on the shooting environment information.

[0015] The operating program of an image processing device relating to the technology of the present disclosure is an operating program of an image processing device equipped with a processor, and causes the processor to perform processing including obtaining shooting environment information for periods before and after the timing of the actual shooting of an image including at least one of a still image or a video, and performing related processing related to the actual image, which is the image that was actually shot, based on the shooting environment information.

[0016] According to the technology of the present disclosure, it is possible to perform related processing related to a real image that has been captured more appropriately than before.

[0017] 1 is a diagram illustrating an overview of an example of use of an image processing device. FIG. 1 is a diagram illustrating an example of a hardware configuration constituting an image processing device. FIG. 2 is a diagram illustrating an overview of information collection processing of an image processing device. FIG. 3 is a diagram illustrating an example of timing for acquiring a main image and shooting environment information. A flowchart showing the procedure of related processing. FIG. 4 is a diagram illustrating an example of a main image and related audio. FIG. 5 is a diagram illustrating an example of album creation, which is an example of related processing. FIG. 6 is a diagram illustrating an example of a summary video, which is an example of related processing. FIG. 7 is a diagram illustrating an example of caption insertion, which is an example of related processing. FIG. 8 is a diagram illustrating an example of image generation, which is an example of related processing. FIG. 9 is a diagram illustrating an example of using a live view image as shooting environment information. FIG. 10 is a diagram illustrating an example of processing for reducing related audio. FIG. 11 is a diagram illustrating an example of text conversion for reducing related audio. FIG. 12 is a diagram illustrating an example of acquiring shooting environment information by a device different from a camera. FIG. 13 is a diagram illustrating an example of using a search history and a movement history as shooting environment information. FIG. 14 is a diagram illustrating an example of using the search history and the movement history to insert a caption.

[0018] 1 is an example of a camera according to the technology of the present disclosure, and is also an example of an image processing device according to the technology of the present disclosure. The smartphone 10 has a function to acquire not only an original image 11 captured by original photography, but also shooting environment information 12 for periods before and after the timing of the original photography. The smartphone 10 also has a function to execute related processing for the original image 11 based on the shooting environment information 12.

[0019] Fig. 1 shows a situation in which a photographer H is using a smartphone 10 to take a picture of a baby BA standing and walking while being watched over by his father FA and mother MA. When the camera function of the smartphone 10 is turned on, a live view image and a shutter button 10A are displayed on the screen. A main image 11 is taken by the photographer H operating the shutter button 10A. In Fig. 1, the smartphone 10 has taken images P1, P2, and P3 as the main image 11.

[0020] Here, "main photography" refers to the process of capturing an image and recording the captured image in a manner that allows playback after the capture is complete. The main image 11 refers to an image that includes at least one of a still image and a video acquired by the main photography. If the main image 11 is a still image, it refers to an image that is captured by operating the shutter button 10A and recorded in storage 27, such as an internal memory or an external memory, in a manner that allows playback after the capture. If the main image 11 is a video, it refers to an image that is recorded from the time an instruction to start shooting is input by operating the shutter button 10A until an instruction to stop shooting is input.

[0021] Live view images are not included in the main image 11. This is because, regardless of the operation of the shutter button 10A, shooting of live view images begins when the camera function is turned on, and after being displayed on an LCD display or the like, they are sequentially erased. In this way, the process of recording live view images is not started based on the intention of the photographer H, such as by operating the shutter button 10A, and is not a process of recording captured images so that they can be played back after shooting is completed. Therefore, live view images are not included in the main image 11, and the process of recording live view images is not included in the main shooting.

[0022] 1, the smartphone 10 is provided with a microphone 29, and collects related audio 12S as the imaging environment information 12 through the microphone 29. The imaging environment information 12 is information generated in periods before and after the timing of the actual imaging of the main image 11. Typically, the information is generated at the location where the actual imaging is performed, and is information that has a spatial relevance in addition to a temporal relevance to the actual imaging.

[0023] The related sounds 12S are sounds generated in the shooting environment where the subject is present during a period before and after the timing of the actual shooting, and have a temporal and spatial relationship with the actual shooting. In the example shown in Figure 1, the subject baby BA is standing and walking while making inarticulate sounds such as "Ah, ah," and the father FA, who is watching over the baby BA from the surrounding area, is yelling at the baby BA, saying "Don't fall!" and the mother MA, saying "You're in a good mood!" As will be described later, the smartphone 10 performs various related processes related to the actual image 11 based on the shooting environment information 12 collected in this way.

[0024] 2 , the smartphone 10 includes a display unit 21, an input unit 22, a processor 23, a memory 24, a communication I / F (Interface) 25, a storage 27, and a camera 28. In addition, the smartphone 10 includes a microphone 29 that collects audio such as the related audio 12S, and a speaker 30 that outputs audio. These components are interconnected via a bus line 26.

[0025] The display unit 21 is a display that displays the original image 11, a live view image, and an operation screen. The input unit 22 is operation buttons and the like for inputting operation instructions. When the display unit 21 is configured as a touch panel display, the touch panel display also constitutes the input unit 22. The shutter button 10A is an example of the input unit 22. The storage 27 is a data storage device such as a flash memory. The memory 24 is a work memory for the processor 23 to execute processing, and is constituted by a RAM (Random Access Memory) and the like.

[0026] The storage 27 stores the original image 11 and the shooting environment information 12. The storage 27 also stores control programs such as an operating system, various application programs, and various data associated with these programs. The processor 23 is, for example, a CPU (Central Processing Unit). The processor 23 loads the programs stored in the storage 27 into the memory 24 and executes processing in accordance with the programs, thereby providing overall control of each part of the computer. The application programs included in the programs are an example of operating programs for the image processing device according to the technology of the present disclosure. The processor 23 functions as an image processing device by executing the application programs. The application programs also include programs that execute various related processes, which will be described later, and by executing these programs, the processor 23 functions as a processing unit that executes the related processes.

[0027] The communication I / F 25 is a network interface for communication via a communication network and performs transmission control. The camera 28 is a built-in digital camera.

[0028] As shown in FIG. 3 , the processor 23 is capable of executing information collection processing and related processing. The information collection processing is processing for collecting shooting environment information 12, such as related audio 12S. When the camera function is activated and the information collection function is turned on in the smartphone 10, the processor 23 collects the shooting environment information 12. In a case where still images such as P1, P2, and P3 and videos such as M1, M2, and M3 are recorded as the main image 11, if the information collection function is turned off, only the main image 11 is recorded. On the other hand, if the information collection function is turned on, the shooting environment information 12 is recorded in addition to the main image 11.

[0029] 4 shows an example of a timing chart, and when the camera function is activated on the smartphone 10, acquisition of a live view image begins. As the shooting mode, a still image shooting mode or a video shooting mode is selected, and in each shooting mode, it is selected whether the information collection function is turned on or off.

[0030] In the still image shooting mode, when an actual shooting instruction (indicated by "ON" in FIG. 4) is input by operating the shutter button 10A, actual images 11 such as P1, P2, and P3 taken at that timing are recorded. While the information collection function is on in the still image shooting mode, collection of related sounds 12S continues as shooting environment information 12, regardless of whether an actual shooting instruction is given or not.

[0031] In video shooting mode, video recording is started by operating the shutter button 10A to issue a main shooting instruction (indicated by "START" in FIG. 4), and video recording continues until an end instruction is input. As a result, videos such as M1, M2, and M3 are recorded as the main image 11. Even in video shooting mode, as long as the information collection function is on, collection of related audio 12S continues as shooting environment information 12. In video shooting mode, when recording of the video that is the main image 11 is started by a main shooting instruction, collection of related audio 12S is stopped, and when recording of the video that is the main image 11 is completed, collection of related audio 12S is resumed. Even in video shooting mode, when the information collection function is off, only recording of the video that is the main image 11 is performed.

[0032] In the video shooting mode, the related audio 12S may be collected in parallel with the video recording. Of course, the audio recorded in the video overlaps with the related audio 12S, so from the viewpoint of reducing the amount of data to be saved, it is preferable to stop collecting the related audio 12S while the video is being recorded, as in this example.

[0033] The original image 11 and the photographing environment information 12 are time stamped so that the chronological relationship between the original image 11 and the photographing environment information 12 can be grasped.

[0034] Returning to FIG. 3 , the related process is, for example, an image editing process for editing the master image 11. Image editing processes include, for example, album creation and video editing. Album creation is a process for selecting some still images from the master image 11 and creating an album. Video editing is a process for extracting a portion of the video of the master image 11 and creating a summary video that summarizes highlight scenes or a shortened short video. Note that in album creation, this example describes an example in which, when the master image 11 is a plurality of still images, some still images are selected from the plurality of still images. However, when creating an album, when the master image 11 is a video, still images for the album may be selected from a plurality of frame images that make up the video.

[0035] 5, when the shooting environment information 12 has been collected, the smartphone 10 executes related processing related to the original image 11 based on the shooting environment information 12. In step ST1100, the processor 23 waits for an instruction to start the related processing. If the instruction to start has been input (Y in step ST1100), the processor 23 proceeds to step ST1200. The instruction for the related processing includes a designation of the original image 11 to be processed.

[0036] In step ST1200, the processor 23 determines whether or not the shooting environment information 12 for a period before and after the actual shooting of the main image 11 to be processed has been recorded. If the shooting environment information 12 is available (Y in step ST1200), the processor 23 proceeds to step ST1300. In step ST1300, the processor 23 acquires the shooting environment information 12 for a period before and after the actual shooting of the main image 11 to be processed. Then, in step ST1400, the processor 23 executes the related processing based on the shooting environment information 12. On the other hand, if the shooting environment information 12 is not available (N in step ST1200), the processor 23 proceeds to step ST1500, where the processor 23 executes the related processing without using the shooting environment information 12.

[0037] The related processing will be specifically described using an example of an album creation process with reference to Figures 6 to 8. Figure 6 more specifically illustrates the photographic scene illustrated in Figure 1. As shown in Figure 6, images P1, P2, and P3, with baby BA as the subject, are recorded in this order as the main image 11. P1 and P2 are scenes of baby BA standing and walking cheerfully, with baby BA smiling. In P2, father FA and mother MA are also captured, watching over baby BA with smiles on their faces. In the periods before and after P1 and P2, sounds uttered by father FA and mother MA, such as "Happy!" and "Don't fall!", are recorded in the related audio 12S.

[0038] In P3, which follows P2, the smiling face of P2 is suddenly replaced by baby BA crying loudly, and father FA is also seen supporting baby BA from behind to soothe him. In the period prior to P3, related audio 12S records father FA's voice saying, "Look, he fell!" and baby BA's crying, "Eeek, eeek." This related audio 12S reveals that baby BA fell during the period between P2 and P3, but the scene of baby BA falling itself was missed and is not recorded as main image 11.

[0039] 7 shows an example of album creation as related processing, assuming the existence of the original image 11 and related audio 12S as the shooting environment information 12. The processor 23 functions as a processing unit that executes album creation by executing an application program for album creation. In the album creation processing, the processor 23 executes a process of automatically selecting images to be included in the album from the original images 11. The image selection is executed by analyzing the image content based on image recognition processing. For example, through the image recognition processing, the processor 23 determines the image content, such as that for P2 and P3, baby BA is the main subject, that the face of baby BA is smiling in P2, and that the face of baby BA is crying in P3.

[0040] Then, when there is related audio 12S as the shooting environment information 12, the processor 23 uses the related audio 12S to select an image. One example of the application program is a program that uses a machine learning model.

[0041] In the absence of related audio 12S, processor 23 selects images to be used in the album based solely on the analysis results of the image content. In general selection criteria, for example, when the subject is a person, images that show a smiling face are often highly rated as album images, while images with crying faces are often relatively poorly rated. In album creation application programs, when image selection is performed using only the analysis results of the image content, training is often performed to ensure that image selection is performed according to such general selection criteria. Therefore, in the absence of shooting environment information 12, processor 23 selects images according to such general selection criteria, excluding images of crying baby BA such as P3 and selecting smiling baby P2.

[0042] On the other hand, when the related audio 12S is included as the shooting environment information 12, the album creation application program is trained to select images taking into consideration the analysis results of the related audio 12S in addition to the analysis results of the image content. In the sequence of scenes shown in FIG. 6 , in which baby BA is happily standing and walking, but then falls and bursts into tears, judging from the smiling expression of father FA in P3, baby BA's fall is not a serious situation, and father FA and mother MA appear to find baby BA's fall rather amusing. Therefore, baby BA's crying face is also a fond memory for father FA and mother MA, and an image of crying face like P3 should be selected as a memorable image for the album. The album creation application program recognizes the sequence of events based on the analysis results of the related audio 12S in addition to the analysis results of the image content, and is trained to increase the evaluation value of images like P3.

[0043] By executing this application program, the processor 23 acquires the related audio 12S from the period before and after each capture to understand the image content of P1, P2, and P3. The processor 23 then performs audio analysis of the related audio 12S. Based on the sound "Look, I fell!" and the baby BA's cries of "Eeek, Eeek" before P3, the processor 23 determines that the cause of the baby BA's crying face in P3 is the baby's own fall. Furthermore, based on the smiling expression of the father FA of P3, the processor 23 assigns a high evaluation value to the image P3, which shows the crying face of the baby BA, as an image that should be included in the album as a fond memory for the father FA and the mother MA, based on the analysis results of the image content of P1 to P3 and the related audio 12S. As a result, in addition to the smiling image P2, the crying image P3 is also selected as an image for the album.

[0044] Figure 7 shows an example of a process for extracting some still images from a plurality of still images that are main images 11 as images for an album, but as shown in Figure 8, the shooting environment information 12 can also be used when creating a summary video of a video in which the main images 11 are moving images.

[0045] In Fig. 8, the videos M1 and M2 recorded as main images 11 are recordings of a portion of the series of scenes shown in Fig. 6. The video M1 includes a scene S12 in which baby BA is standing and walking happily, as shown in P2 of Fig. 7, and the video M2 includes a scene S13 in which baby BA is crying loudly, as shown in P3 of Fig. 7. However, as in the example of Fig. 7, no image of the scene in which baby BA falls is recorded, and only that scene is recorded as related audio 12S.

[0046] When creating a summary video of M1 and M2, the processor 23 extracts some highlight scenes from M1 and M2. As described in FIG. 7 , if there is no shooting environment information 12, the processor 23 extracts highlight scenes based only on the analysis results of the video. Therefore, the processor 23 extracts a scene S12 in which the baby BA is smiling and excludes a scene S13 in which the baby BA is crying, according to general selection criteria. On the other hand, if there is related audio 12S, the processor 23 can determine that the crying scene S13 is a memorable scene, and therefore extracts the scene S13 as a highlight scene. Note that a highlight scene is not limited to a short video extracted from a video, but may also be a single still image extracted from a frame image constituting the video.

[0047] As described above, the smartphone 10, which is an example of an image processing device related to the technology of the present disclosure, is equipped with a processor 23, and the processor 23 acquires, as an example, from the storage 27, shooting environment information 12 for a period before and after the timing of the actual shooting of an image including at least one of a still image or a video, and performs related processing (as an example, creating an album and creating a summary video) related to the actual image 11, which is the actual shot image, based on the shooting environment information 12.

[0048] Therefore, it is possible to more appropriately perform related processing related to the actually captured main image 11 than in the past. For example, as shown in the above embodiment, image P3 of the crying baby BA shown in Fig. 7 is unlikely to be selected as an image for an album or a highlight scene for a digest video according to the above-mentioned general selection criteria based only on the analysis results of the image content.

[0049] However, when the actual shooting environment is comprehensively judged, an image such as P3 is an image that captures a memorable scene and is considered to be selected as an image for the album or a highlight scene for the digest video. By performing related processing based on the related audio 12S as the shooting environment information 12, the processor 23 can select an image such as P3 as an image for the album or a video including an image such as P3 as a highlight scene for the digest video.

[0050] 7 and 8, the reason why baby BA in the image P3 is crying is not recorded in the actual image 11. Even in such a situation, related processes such as creating an album or a summary video are performed based on the shooting environment information 12 for periods before and after the timing of the actual shooting, making it easy to understand why baby BA in the image P3 is crying. As a result, the processor 23 can appropriately grasp the situation in accordance with the actual shooting environment and execute appropriate related processes.

[0051] In this way, the image processing device according to the technique of the present disclosure can perform appropriate related processing compared to the related processing performed in the past based only on the analysis results of the image content.

[0052] In the above embodiment, the related audio 12S from periods before and after the actual shooting timing is used as the shooting environment information 12. Audio information often expresses subtle nuances of the shooting environment that do not appear in the image. Therefore, using the related audio 12S as the shooting environment information 12 enables more appropriate related processing.

[0053] In the above embodiment, the related process is, as exemplified by the creation of an album and a summary video, a process of extracting at least a portion of the original image 11. By utilizing the technology of the present disclosure in an extraction process such as selecting images for an album or extracting scenes from a summary video, it becomes possible to select a more appropriate extraction target.

[0054] Although the extraction process of the original image 11 has been exemplified by the creation of an album and a summary video, the extraction process may also be the determination of the trimming range when trimming one original image 11 .

[0055] Furthermore, in creating an album, the photographing environment information 12 may be used to determine the layout of the extracted images in addition to the extraction process of the original images 11. For example, the processor 23 may perform processing such as arranging the original image 11 that is closest to the most exciting moment in the related audio 12S with the most laughter in the center of the album so that it stands out more than the other images, or displaying it in a larger size than the other original images 11.

[0056] (Variation of Related Processing: Addition of Additional Information) Furthermore, as shown in Fig. 9, the related processing is not limited to extraction processing, but may be additional information addition processing that adds additional information to be added to the main image 11 based on the shooting environment information 12. Fig. 9 shows, as an example of additional information addition processing, processing that inserts a caption 36 such as an explanatory text according to the content of the shooting scene of the main image 11.

[0057] In Figure 9, the main image 11 and the shooting environment information 12 are, similarly to Figure 7, the main image 11 and the associated audio 12S representing the series of situations shown in Figure 6. In this case, consider an example in which the processor 23 generates a caption 36 to be inserted into an image P3 of a crying baby BA. In the example of Figure 6, there is no image remaining of the scene in which the baby BA falls. Therefore, without the associated audio 12S, it is unclear why the baby BA is crying in the image P3. Therefore, without the associated audio 12S, the processor 23 generates a caption 36B for the image P3, asking the reason for the crying, such as "Why are you crying?", and assigns this to P3.

[0058] On the other hand, if there is related audio 12S before the timing of P3's capture, the processor 23 can determine that the reason baby BA is crying is because he fell down, based on the audio of father FA and mother MA saying, "He fell down!" Therefore, the processor 23 generates a caption 36A indicating the reason for the crying, such as "He fell down!", and assigns this to P3.

[0059] In this way, by using the shooting environment information 12 for the related processing of adding additional information such as inserting a caption, the processor 23 can add appropriate additional information that is more relevant to the original image 11 and is in line with the shooting environment.

[0060] 9, the supplementary information is a caption 36A indicating the reason why baby BA is crying, but this is an example of information that cannot be read from the image 11 alone, and other information may be used. For example, the processor 23 may convert the atmosphere or emotions of people around baby BA from the related audio 12S into text, such as "It's amazing how quickly he got up even after falling!", and use this as supplementary information.

[0061] (Variation of Related Processing: Image Generation) Furthermore, as shown in FIG. 10 , the related processing may be processing for generating a related image related to the main image 11, which is an image from the period when the shooting environment information 12 is acquired, based on the shooting environment information 12. In FIG. 10 , it is assumed that the main image 11 and related audio 12S similar to those in FIGS. 7 and 9 are recorded. In this case, the example of the related image shown in FIG. 10 is an image of the scene where baby BA falls, which was missed in FIG. 6. Based on the related audio 12S, the processor 23 generates, as a related image, P2-1, which is an image of the scene where baby BA falls. This makes it possible to complement important scenes, for example, as images in an album. Generative AI (Artificial Intelligence), an example of a machine learning model, is used to generate the related image.

[0062] Furthermore, instead of generating a new image, the original image 11 may be modified. For example, if the scene of the fall is recorded as the original image 11 but part of the baby BA's body is not captured, the missing part of the body in the image is supplemented by the generation AI, and the image is modified to one that shows the whole body of the baby BA.

[0063] By generating an image related to the main image 11 based on the shooting environment information 12 in this way, it is possible to generate an important scene that was missed in the main shooting.

[0064] (Modification of Shooting Environment Information: Related Image) In the above embodiment, the shooting environment information 12 has been described using the related audio 12S as an example. However, the shooting environment information 12 is not limited to the related audio 12S, and may instead be a live view image 12LV as shown in FIG. 11 . The live view image 12LV is an example of a related image serving as the shooting environment information 12 according to the technology of the present disclosure. As described above, the live view image is not the main image 11, but is information that is normally deleted sequentially. This live view image is saved without being deleted while the information collection function is on, and can be used by the processor 23 as the shooting environment information 12. In this way, it can be used instead of the related audio 12S. Of course, since the live view image requires a large amount of data, the related audio 12S is preferable from the perspective of reducing the amount of data.

[0065] 12 and 13 , the processor 23 may store the shooting environment information 12 in a storage format with reduced data volume based on the analysis results of the content of the shooting environment information 12. The shooting environment information 12 includes information about the period before and after the actual shooting, and therefore the amount of data tends to become enormous. Therefore, the processor 23 may execute a stored data reduction process on the shooting environment information 12 to reduce the amount of stored data.

[0066] 12 shows an example in which a portion of the related audio 12S as the shooting environment information 12 is deleted and the deleted related audio 12SA is stored as saved data. The portion of the related audio 12S to be deleted is, for example, a silent portion that does not contain any audio. Furthermore, if the time interval between the capture timings of two still images included in the main image 11 is long, the processor 23 may perform processing such as deleting a portion that is considered to have little relevance to the main image 11.

[0067] 13 shows an example in which the processor 23 converts the related speech 12S into text and stores the text data 12T. By converting speech data into text, the amount of data can be significantly reduced. Naturally, the converted text data is provided with a timestamp so that the time series can be determined.

[0068] In the above embodiment, an example has been described in which both the original image 11 and the shooting environment information 12 are acquired by the smartphone 10 functioning as a camera. However, as shown in FIG. 14 , the shooting environment information 12 may be acquired by a device other than the camera that captures the original image 11. In the example shown in FIG. 14 , the original image 11 is captured by a camera 40 that is a dedicated camera, and the shooting environment information 12 is collected by the smartphone 10. The captured original image 11 and the collected shooting environment information 12 are associated with each other as related data. As an example of the association, the camera 40 and the smartphone 10 are registered as related devices, and when the smartphone 10 records the shooting environment information 12, the smartphone 10 communicates with the camera 40 and records the file name of the original image 11 in association with the shooting environment information 12. The chronological order of the original image 11 and the shooting environment information 12 can be determined by the timestamps attached to each of them.

[0069] 14, the different device that acquires the image capture environment information 12 is the smartphone 10. However, when the original image 11 is recorded by the camera 40, an accessory attached to the camera 40 may be the different device that acquires the image capture environment information 12. Examples of accessories include a 360° camera and a microphone that are attached to the accessory shoe of the camera 40.

[0070] By recording the image capturing environment information 12 in a device other than the camera in this way, it is possible to reduce the load on resources such as the recording capacity and battery capacity of the camera.

[0071] (Variations of shooting environment information: search history, movement history) Furthermore, the shooting environment information 12 is not limited to related audio 12S and related images (for example, live view images LV), but may also include at least one of the user's movement history 12HM related to the actual shooting and the user's search history 12HS related to the actual shooting.

[0072] The example shown in Fig. 15 is an example in which a user's search history 12HS and a user's movement history 12HM are used as the shooting environment information 12. In the example shown in Fig. 15, similar to the example shown in Fig. 14, the main image 11 is captured by a camera 40, and the shooting environment information 12 is collected by a smartphone 10. However, in the example shown in Fig. 15, the shooting environment information 12 is not the related audio 12S, but the user's search history 12HS and the user's movement history 12HM.

[0073] Here, the user refers to a person present in the shooting environment of the original image 11, specifically, the photographer who shoots the original image 11, the person who is the subject of the original image 11, and any person who is with the photographer and the subject in the shooting environment. The search history 12HS is a history of information searches performed by any of these users. The movement history 12HM is a movement history recorded on the smartphone 10 of any of these users.

[0074] In the example shown in Figure 15, the main image 11 includes images P6 to P10, which show a cherry blossom viewing party. P6 is an image of people reserving a spot for cherry blossom viewing, and P7 and P8 are images of a close-up of cherry blossoms and an entire cherry tree. P9 is an image of the moment of a toast, and P10 is an image of a party taking place while enjoying cherry blossom viewing. The main image 11 is an image that shows a series of such cherry blossom viewing scenes.

[0075] The search history 12HS is a history of information searches performed in the period before the capture of the main image 11, and the search history 12HS may record, for example, a search history of famous cherry blossom viewing spots. The movement history 12HM may record, for example, a history of visits to Chidorigafuchi, one of the famous cherry blossom viewing spots in Tokyo.

[0076] 16 shows an example of the processor 23 creating an album as a related process, assuming that the original image 11 and the shooting environment information 12 have been collected. The processor 23 determines that the main theme of the original image 11 is cherry blossom viewing based on the analysis results of the image content of the original image 11 as well as the search history 12HS and movement history 12HM for the period prior to the shooting timing of the original image 11. The processor 23 then selects many cherry blossom viewing images in line with the main theme of cherry blossom viewing. Furthermore, if the original image 11 contains few flower images, for example, the processor 23 generates flower images and adds the generated images.

[0077] The search history 12HS and movement history 12HM may also be used as shown in Figures 17 and 18. In the example shown in Figure 17, in addition to the cherry blossom viewing images P6 to P10 shown in Figure 15, an image P11 of people eating at a Chinese restaurant and an image P12 of a crying child covering his mouth are recorded as main images 11. In addition, between P10, the last image of the cherry blossom viewing, and P11, the image showing the dining scene at the restaurant, a search history 12HS of a search for a nearby restaurant and a movement history 12HM of a movement to the Chinese restaurant are collected.

[0078] 18 illustrates an example in which the processor 23 inserts a caption into the image P12 as a related process, assuming that the original image 11 and the shooting environment information 12 have been collected. The processor 23 determines the image P12 as follows, based on the analysis results of the image content of the original image 11, as well as the search history 12HS and movement history 12HM for the period before and after the capture of the original image 11. That is, the processor 23 determines that the image P12 was taken while eating at a Chinese restaurant visited after cherry blossom viewing. Furthermore, based on the analysis results of the image content, such as the child crying and covering his mouth, the processor 23 determines that the image shows the child having suffered a minor burn while eating Chinese food. Based on these analysis results, the processor 23 then generates a caption 36 such as "I burned my lip while eating Chinese food after cherry blossom viewing" and assigns it to the image P12.

[0079] In this way, by using the user's search history 12HS and movement history 12HM as the shooting environment information 12, it may be effective for various related processes on the original image 11.

[0080] Of course, the above-mentioned related processes may be selectively executed in a number of types, such as album creation and caption insertion.

[0081] In the above example, the image editing is exemplified by the creation of an album, but it is also possible to create a calendar or the like instead of an album.

[0082] If the main image 11 contains a plurality of subjects, the images may be classified by subject.

[0083] The above description can be used to understand the technologies described in the following supplementary items. [Supplementary Item 1] An image processing device including a processor, wherein the processor acquires shooting environment information for periods before and after a timing of actual shooting of an image including at least one of a still image or a video, and executes related processing related to the actual image, which is the image that was actually shot, based on the shooting environment information. [Supplementary Item 2] The image processing device described in Supplementary Item 1, wherein the shooting environment information includes related audio for the period. [Supplementary Item 3] The image processing device described in Supplementary Item 1 or Supplementary Item 2, wherein the shooting environment information includes related images for the period. [Supplementary Item 4] The image processing device described in any one of Supplementary Items 1 to 3, wherein the related processing is processing to extract at least a portion of the actual image. [Supplementary Item 5] The image processing device described in any one of Supplementary Items 1 to 4, wherein the related processing is processing to add related information to be attached to the actual image based on the shooting environment information. [Supplementary Item 6] The image processing device according to any one of Supplementary Items 1 to 5, wherein the related processing is processing for generating, based on the shooting environment information, a related image related to the main image during the period in which the shooting environment information is acquired. [Supplementary Item 7] The image processing device according to any one of Supplementary Items 1 to 6, wherein the processor saves the shooting environment information in a storage format with reduced data volume based on the results of analyzing the contents of the shooting environment information. [Supplementary Item 8] The image processing device according to any one of Supplementary Items 1 to 7, wherein the shooting environment information is acquired by a device different from the camera that captures the main image. [Supplementary Item 9] The image processing device according to any one of Supplementary Items 1 to 8, wherein the shooting environment information includes at least one of a user's movement history related to the main shooting and a user's search history related to the main shooting. [Supplementary Item 10] A method for operating an image processing device having a processor, wherein the processor acquires shooting environment information for periods before and after the timing of actual shooting of an image including at least one of a still image or a video, and executes related processing related to the actual image, which is the image that was actually shot, based on the shooting environment information.[Supplementary Item 11] An operating program for an image processing device having a processor, the operating program causing the processor to execute processes including: acquiring shooting environment information for periods before and after the timing of actual shooting of an image including at least one of a still image or a video; and executing related processes related to the actual image, which is the image that was actually shot, based on the shooting environment information.

[0084] In the above-described embodiments, the processes performed by the processing unit that executes the related processes of the image processing device are executed by an arbitrary computer. Furthermore, the arbitrary computer may execute these processes by a processor as hardware, a program as software, or a combination thereof. In such a case, the processor is configured to execute the various processes in the present embodiment in cooperation with the program, and may function as each unit or means in the present embodiment. Furthermore, the order in which the processes are executed by the processor is not limited to the order described above and may be changed as appropriate.

[0085] The given computer may be a general-purpose computer, a computer for specific applications, a workstation, or any other system capable of executing each process. The processor may be configured with one or more pieces of hardware, and the type of hardware is not limited. For example, the processor may be configured with hardware such as a central processing unit (CPU), a micro processing unit (MPU), a programmable logic device such as a field programmable gate array (FPGA), a dedicated circuit for executing specific processes such as an application specific integrated circuit (ASIC), a graphics processing unit (GPU), or a neural processing unit (NPU). The type of hardware may also be a combination of different types of hardware. When multiple pieces of hardware are configured to execute one or more processes of a certain processor, the multiple pieces of hardware may be located in devices physically separated from each other, or may be located in the same device. Furthermore, in any embodiment, the order of the processes performed by the processor is not limited to the order described above and may be changed as appropriate. The hardware is configured by an electric circuit (circuitry) or the like that combines circuit elements such as semiconductor elements.

[0086] Furthermore, the program may be software, such as firmware or microcode. The program may also be, for example, a group of program modules, each function of which may be implemented by a processor configured to perform the respective function. The program may be program code and / or multiple code segments stored in one or more non-transitory computer-readable media (e.g., storage media and / or other storages). The program may be stored across multiple non-transitory computer-readable media that reside in physically separate devices. The program code or code segment may represent a procedure, a function, a subprogram, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements. The program code or code segment may be connected to another code segment or a hardware circuit by sending or receiving information, data, arguments, parameters, or memory contents.

[0087] The technology of the present disclosure can also be appropriately combined with the various embodiments and / or various modified examples described above. Furthermore, the technology is not limited to the above embodiments, and various configurations can be adopted without departing from the spirit of the present disclosure. Furthermore, the technology of the present disclosure also covers, in addition to programs, storage media that non-temporarily store programs. The storage medium is, for example, a computer-readable non-temporary storage medium such as a USB (Universal Serial Bus) memory, a flexible disk, or a CD-ROM (Compact Disc Read Only Memory). The program may also be provided online via a network such as the Internet. The technology of the present disclosure also covers, in addition to programs, program products. A program product includes any type of product for providing a program. Like a program, a program product may be provided stored on a computer-readable non-temporary storage medium or provided online.

[0088] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[0089] In this specification, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0090] The disclosure of Japanese Patent Application No. 2024-057946, filed on March 29, 2024, is incorporated herein by reference in its entirety. In addition, all documents, patent applications, and technical standards described herein are incorporated herein by reference to the same extent as if each individual document, patent application, and technical standard was specifically and individually indicated to be incorporated by reference.

Claims

1. An image processing device comprising a processor, which acquires shooting environment information for periods before and after the timing of actual shooting of an image including at least one of a still image and a video, and executes related processing related to the actual image, which is the image that was actually shot, based on the shooting environment information.

2. The image processing device according to claim 1, wherein the shooting environment information includes related audio for the period.

3. The image processing device according to claim 1, wherein the shooting environment information includes related images for the period.

4. The image processing device according to claim 1, wherein the related processing is processing for extracting at least a part of the original image.

5. The image processing device according to claim 1, wherein the related processing is an additional information adding processing for adding additional information to the main image based on the shooting environment information.

6. The image processing device according to claim 1, wherein the related processing is processing for generating a related image related to the main image during the period when the shooting environment information is acquired, based on the shooting environment information.

7. The image processing device according to claim 1, wherein the processor stores the shooting environment information in a storage format that reduces the amount of data based on the analysis result of the content of the shooting environment information.

8. The image processing device according to claim 1, wherein the shooting environment information is acquired by a device different from the camera that shoots the main image.

9. The image processing device according to claim 1, wherein the shooting environment information includes at least one of a user's movement history related to the actual shooting and a user's search history related to the actual shooting.

10. A method for operating an image processing device equipped with a processor, wherein the processor acquires shooting environment information for periods before and after the timing of actual shooting of an image including at least one of a still image or a video, and executes related processing related to the actual image, which is the image that was actually shot, based on the shooting environment information.

11. An operating program for an image processing device equipped with a processor, which causes the processor to perform processing including: acquiring shooting environment information for periods before and after the timing of actual shooting of an image including at least one of a still image or a video; and executing related processing related to the actual image, which is the image actually shot, based on the shooting environment information.

Images