Information processing device, information processing method, and program

Abstract

The present invention provides an information processing device, method, and program that enable both a reduction in image quality degradation and a reduction in processing load. [Solution] The information processing device detects a predetermined object from an input image, such as a video or still image, generates a background image after extracting the image of the predetermined object, and further generates text data representing the features of the background image using neural network processing or the like. The information processing device then saves or transmits the image of the detected object and the text data representing the features of the generated background image. It also generates a background image based on the text data and reconstructs the image by superimposing the image of the object onto the background image.

Description

Technical Field

[0001] The present invention relates to information processing technology based on images.

Background Art

[0002] In recent years, image generation technologies have been developed that use a Visual Language Model (VLM) and a diffusion model to generate images by inputting a sentence called a prompt, such as DALL·E2 developed by OpenAI. Further, Patent Document 1 discloses a method of outputting the result of recognition processing by a trained Deep Neural Network (DNN) with an image as input.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the method described in Patent Document 1, when generating an image using a VLM and a diffusion model with the recognition processing result as a prompt, only the reproduction of the recognized object, that is, the reproduction of the subject can be achieved, and the background image is lost and the image quality deteriorates. Furthermore, when generating continuous images such as a video, recognition processing is performed on a plurality of images in a short time, increasing the processing load.

[0005] Therefore, an object of the present invention is to suppress deterioration of image quality and also reduce the processing load.

Means for Solving the Problems

[0006] The present invention comprises a detection means for detecting a predetermined object from an input image and generating an image of the object and a background image, and a text generation means for generating text data representing the features of the background image, wherein the image of the object detected by the detection means and the text data generated by the text generation means are stored or transmitted. [Effects of the Invention]

[0007] According to the present invention, it is possible to suppress the degradation of image quality and reduce the processing load. [Brief explanation of the drawing]

[0008] [Figure 1] This figure shows an example configuration of an information processing device that converts background images into text data. [Figure 2] This is a flowchart for information processing that converts a background image into text data. [Figure 3] This is an explanatory diagram illustrating the generation of the main subject image and the conversion of the background image into text data. [Figure 4] This is an explanatory diagram of the equipment movement information. [Figure 5] This figure shows an example configuration of an information processing device for restoring images. [Figure 6] This is a flowchart for image restoration processing. [Figure 7] This is an explanatory diagram for background image generation and image restoration. [Modes for carrying out the invention]

[0009] Embodiments of the present invention will be described below with reference to the drawings. The following embodiments are not limiting to the present invention, and not all of the features described in the embodiments are essential to the solution of the present invention; these features may be combined arbitrarily. The configuration of the embodiments may be modified or changed as appropriate depending on the specifications of the apparatus to which the present invention is applied and various conditions (operating conditions, operating environment, etc.). In addition, in the following embodiments, redundant explanations of the same or similar configurations or processing steps will be omitted.

[0010] This embodiment describes an information processing device that generates text data based on a background image obtained by extracting an image of a predetermined object from an input video or still image, and saves (records) or transmits the text data and the image of the predetermined object. Furthermore, this embodiment also describes an information processing device that generates a background image using the saved or transmitted text data, and superimposes the image of the predetermined object onto the generated background image to reconstruct an image in which the predetermined object exists within the background image. In this embodiment, an image captured is given as an example of an input image, and a subject such as a person is given as an example of an image of a predetermined object within the image captured. Note that the input image is not limited to an image captured, but may be, for example, an image captured from a separately created video or a video on a network, or an image generated by CG technology or generation AI technology.

[0011] Figure 1 is a block diagram showing the schematic configuration of an information processing device 100 that generates text data usable as a prompt based on a background image obtained by extracting the image of the main subject from an captured image, and then saves or transmits the text data and the image of the main subject. The information processing device 100 has a configuration in which a CPU 101, non-volatile memory 102, memory 103, imaging unit 105, subject detection unit 106, text generation unit 107, movement information generation unit 108, generation determination unit 109, communication unit 110, etc. are connected via a system bus 104.

[0012] The imaging unit 105 captures an image by converting the light image captured through the lens optical system into an electrical signal, and consists of, for example, a CMOS (Compact Metal Oxide Semiconductor) image sensor and its peripheral circuitry. The image captured by the imaging unit 105 is temporarily stored in the memory 103. The lens optical system may be included in the imaging unit 105 or it may be a detachable lens optical system. In this embodiment, an example is given in which video recording is performed by the imaging unit 105, and the imaging unit 105 outputs an image captured for each frame.

[0013] The subject detection unit 106 reads the captured image frame by frame from the memory 103 and performs calculations on the captured image using the first trained model to detect a predetermined subject, such as a person (hereinafter referred to as the main subject), within the captured image. The first trained model is a pre-trained model that has been trained in advance to detect subjects such as people from an input image. Furthermore, the subject detection unit 106 generates an image of the detected main subject (hereinafter referred to as the main subject image) and an image of the background excluding the main subject (hereinafter referred to as the background image). The subject detection unit 106 also acquires the image size information of the captured image and the position information of the main subject image within the captured image for each frame. Note that the image size information of the captured image does not necessarily need to be acquired for each frame; for example, it may be acquired only for the first frame. The subject detection unit 106 then stores the main subject image for each frame, the position information of the main subject image within the captured image, and the image size information of the captured image in the non-volatile memory 102, and temporarily stores the background image in the memory 103.

[0014] The text generation unit 107 reads the background image temporarily stored in memory 103, performs calculations on the background image using a second pre-trained model, generates text data representing the features of the background image, and stores it in memory 103. The second pre-trained model is a pre-trained model that generates and outputs text data representing the features of an input image. In this embodiment, the text data generated by the text generation unit 107 is used as a prompt when generating the background image in the information processing device 500 shown in Figure 5, which performs image restoration processing described later.

[0015] The movement information generation unit 108 acquires the acceleration and angular velocity corresponding to the movement of the information processing apparatus 100 having the imaging unit 105, and generates apparatus movement information indicating the movement direction and movement amount of the information processing apparatus from those acceleration and angular velocity. In the information processing apparatus 100 of the present embodiment, the imaging unit 105 includes an acceleration sensor and a gyro sensor, and the movement information generation unit 108 acquires the acceleration from the acceleration sensor and the angular velocity from the gyro sensor. Then, the movement information generation unit 108 stores the apparatus movement information in the memory 103.

[0016] The generation determination unit 109 performs determination processing with a predetermined determination condition as to whether the elapsed time after the previous text data generation has exceeded a predetermined time threshold, and determines whether to perform text generation by the text generation unit 107 according to the determination result. For example, when the elapsed time from the previous text data generation exceeds the time threshold, the generation determination unit 109 instructs the text generation unit 107 to newly generate text data. On the other hand, when the elapsed time from the previous text data generation is within the time threshold, the generation determination unit 109 determines that no new text generation is performed, and instructs the movement information generation unit 108 to generate apparatus movement information. Note that the predetermined time threshold for the elapsed time may be the time of one frame, the time for every few frames or every dozens of frames, or the time in seconds or minutes.

[0017] The communication unit 110 transmits the main subject image for each frame, the position information of the main subject, and the apparatus movement information stored in the non-volatile memory 102, as well as the image size information and the text data generated according to a predetermined determination condition, to another information processing apparatus. In the case of the present embodiment, it is assumed that the another information processing apparatus is the information processing apparatus 500 in FIG. 5 that performs the image restoration processing described later. Note that each information from the main subject image stored in the non-volatile memory 102 to the apparatus movement information may be stored in a removable memory card or the like not shown, and may be sent to another information processing apparatus via the removable memory card or the like.

[0018] The CPU 101 controls the operations of each part connected via the system bus 104 by executing the programs stored in the non-volatile memory 102. The non-volatile memory 102 is a non-volatile memory that can be electrically erased and stored, and for example, ROM (Read Only Memory), hard disk, etc. are used. In the non-volatile memory 102, various programs for the operation of the CPU 101, the first learned model used by the subject detection unit 106, the second learned model used by the text generation unit 107, the time threshold of the elapsed time used by the generation determination unit 109, etc. are stored. However, this is just an example, and when the neural network arithmetic processing is not used for the determination of the main subject in the subject detection unit 106, the storage of the first learned model becomes unnecessary. Similarly, when the neural network arithmetic processing is not used for the generation of text data representing the characteristics of the background image in the text generation unit 107, the storage of the second learned model becomes unnecessary. The memory 103 is a rewritable volatile memory, such as DRAM (Dinamic Random Access Memory). The CPU 101 controls the operations of each part by executing the program read from the non-volatile memory 102 and expanded in the memory 103, and also uses the memory 103 as a work memory.

[0019] Note that information processing programs for the CPU 101 to realize the functions of the subject detection unit 106, the text generation unit 107, the movement information generation unit 108, and the generation determination unit 109 may be stored in the non-volatile memory 102. In this case, the CPU 101 can realize the functions of the subject detection unit 106, the text generation unit 107, the movement information generation unit 108, and the generation determination unit 109 by executing the information processing program read from the non-volatile memory 102 and expanded in the memory 103.

[0020] Figure 1 shows an example configuration in which the information processing device 100 includes an imaging unit 105, but the imaging unit 105 may be a separate external device (imaging device). If the imaging unit 105 is an external imaging device, the imaging device also includes an acceleration sensor and a gyro sensor, and provides the information processing device with the captured image, as well as acceleration data and angular velocity data indicating the movement of the imaging device during imaging. In this case, the movement information generation unit 108 generates device movement information indicating the direction and amount of movement of the imaging device based on the acceleration data and angular velocity data provided from the imaging device. The captured image, acceleration data, and angular velocity data from the imaging device to the information processing device may be provided via a removable memory or via a communication channel.

[0021] Figure 2 is a flowchart showing the flow of information processing performed in the information processing device 100. First, in step S200, the imaging unit 105 performs video recording and acquires frame-by-frame images. These frame-by-frame images are then temporarily stored in the memory 103 under the control of the CPU 101.

[0022] Next, in step S201, the subject detection unit 106 reads the first trained model from the non-volatile memory 102 and reads the captured image frame by frame from the memory 103. The subject detection unit 106 then performs neural network processing using the first trained model on the captured image frame by frame to detect the main subject, such as a person. Furthermore, the subject detection unit 106 extracts an image of a rectangular region containing the main subject from the captured image frame by frame, and generates a main subject image by applying transparency processing to the image portion of the extracted rectangular region excluding the main subject. The subject detection unit 106 also acquires the remaining image after removing the rectangular region of the main subject from the captured image as a background image. The subject detection unit 106 then acquires the image size information and the position information of the main subject image within the captured image, stores them in the non-volatile memory 102 in association with the main subject image, and temporarily stores the background image in memory 103.

[0023] Next, in step S202, the generation determination unit 109 reads a time threshold from the non-volatile memory 102 and determines whether the elapsed time since the previous text data generation in the text generation unit 107 exceeds the time threshold. For example, if the elapsed time since the previous text data generation exceeds the time threshold, the generation determination unit 109 instructs the text generation unit 107 to generate text data. The generation determination unit 109 also instructs the text generation unit 107 to generate text data if the elapsed time since the previous text data generation is 0 (zero). On the other hand, if the elapsed time since the previous text data generation is within the time threshold, the generation determination unit 109 instructs the movement information generation unit 108 to generate device movement information. In other words, if the generation determination unit 109 determines in step S202 that the elapsed time since the previous text data generation exceeds the time threshold, the CPU 101 proceeds to step S203, which is performed by the text generation unit 107. On the other hand, if the generation determination unit 109 determines that the elapsed time is within the time threshold, the CPU 101 proceeds to step S204, which is performed by the movement information generation unit 108.

[0024] If the process proceeds to step S203, the text generation unit 107 reads the second trained model from the non-volatile memory 102 and reads the background image of the frame instructed by the generation determination unit 109 from the memory 103. The text generation unit 107 then performs neural network processing using the second trained model on the background image to generate text data representing the features of the background image. Subsequently, the text generation unit 107 stores the text data representing the features of the background image in memory 103, linked to the main subject image. The second trained model can pre-set the level of detail of the text data representing the features of the background image, and generates text data with a level of detail corresponding to that setting. However, the higher the set level of detail, the larger the amount of text data representing the features of the background image. An example of text data representing the features of the background image will be described later.

[0025] On the other hand, if the process proceeds to step S204, the movement information generation unit 108 generates device movement information indicating the direction and amount of movement of the information processing device 100 (i.e., the imaging unit 105) from the acceleration and angular velocity indicating the movement of the information processing device 100 having the imaging unit 105. The movement information generation unit 108 then stores the generated device movement information in the memory 103, linked to the main subject image. After step S203 or step S204, in step S205, the CPU 101 determines whether to terminate the process shown in the flowchart in Figure 2, based on, for example, whether or not there is a termination instruction from the user, or whether imaging continues or ends by the imaging unit 105. If the CPU 101 determines that it does not need to terminate, it returns to step S200; if it determines that it needs to terminate, it terminates the process shown in the flowchart in Figure 2.

[0026] In this embodiment, the information processing device 100 performs neural network computation using a second trained model on the background image, as described above, to generate text data that represents the features of the background image. That is, the information processing device 100 in this embodiment does not generate text data that represents the features of the captured image including the main subject, but rather generates text data that represents the features of the background image, limited to the background image. In this embodiment, the text data that represents the features of the background image is used as a prompt during the image restoration process performed by the information processing device 500 in Figure 5, which will be described later. For example, in the method described in Patent Document 1, if an image is generated using VLM and a diffusion model with the recognition processing result as a prompt, as described above, only the recognized object, i.e., the subject, can be reproduced, and the background image is lost, resulting in a decrease in image quality. In contrast, by using a prompt that represents the features of the background image as in this embodiment, the background image can be generated during image restoration, and the decrease in image quality can be suppressed. Furthermore, according to this embodiment, by limiting the neural network computation for generating text data to the background image, it is also possible to reduce the processing load.

[0027] Next, we will explain the generation of the main subject image and background image in step S201 of the flowchart in Figure 2, and the generation of text data representing the characteristics of the background image in step S203, using the captured image shown in Figure 3 as an example. Figure 3(a) shows an example of an image 300 acquired by the imaging unit 105. The image 300 is a daytime image of a sunny day with a lake in a mountainous area as the background 302, and a male figure 301 standing in front of the lake. In this embodiment, if the predetermined subject to be detected by the subject detection unit 106 is, for example, a person, the subject detection unit 106 detects the person 301, who is approximately in the center of the image 300, as the main subject. Then, as shown in Figure 3(b), the subject detection unit 106 cuts out a rectangular region from the image 300 that encloses the person 301. Furthermore, the subject detection unit 106 performs a transparency process on the image of the rectangular region, excluding the main subject person 301, that is, the slightly visible background portion, to generate an image of the person 301 as the main subject image, as shown in Figure 3(c). The subject detection unit 106 also generates a background image as shown in Figure 3(d) from the remaining image of the image 300 after cutting out the rectangular region of the main subject. As a result, the non-volatile memory 102 stores an image of the rectangular area of ​​the main subject, which includes the person 301 shown in Figure 3(c), and the memory 103 stores a background image as shown in Figure 3(d).

[0028] Here, if the generation determination unit 109 determines that the elapsed time since the previous text data generation exceeds a time threshold, the text generation unit 107 generates text data representing the features of the background image stored in memory 103. Figure 3(e) shows an example of text data 303 generated by the text generation unit 107 from the background image in Figure 3(d). In the example in Figure 3(e), text data 303 is generated that expresses in sentences the overall features of the background, such as daytime and sunny weather, the features of three gently sloping mountains, the features of forests, lakes and low grass, and the features of their positional relationships. This text data 303 is used as a prompt when restoring the background image in the information processing device 500 in Figure 5, which performs image restoration processing described later. That is, since the prompt is text data, the amount of data is very small, and it is also possible to use highly compressed lossless compression such as gzip (GNUzip), so the amount of data of the background image can be greatly reduced.

[0029] Next, the generation of device movement information in step S204 of the flowchart in Figure 2 will be explained using the example of the case where the information processing device according to this embodiment is mounted on an imaging device as shown in Figure 4. Here, the horizontal direction of the imaging device equipped with the information processing device of this embodiment will be described as the X-axis and the vertical direction as the Y-axis. The imaging device is equipped with an acceleration sensor to acquire acceleration and a gyro sensor to acquire angular velocity, and the movement information generation unit 108 generates device movement information from these acceleration and angular velocity.

[0030] Figure 4(a) shows the acceleration acquired by the imaging device. Since the imaging device is equipped with an acceleration sensor, the motion information generation unit 108 of the information processing device 100 can acquire the acceleration ax in the X-axis direction and the acceleration ay in the Y-axis direction. Figure 4(b) shows the angular velocity acquired by the imaging device. Since the imaging device is equipped with a gyro sensor, the motion information generation unit 108 of the information processing device 100 can acquire the angular velocity ω of the imaging device in the XY plane. Figure 4(c) shows the device movement information generated by the movement information generation unit 108 of the information processing device 100. The device movement information consists of the amount of movement of the imaging device in the X-axis direction, the amount of movement in the Y-axis direction, and the inclination in the XY plane, which are obtained by multiplying the acquired acceleration ax, ay and angular velocity ω by time t. If t is the time taken for one frame in video recording, the movement information generation unit 108 generates the amount of movement in the X-axis direction ax·t, the amount of movement in the Y-axis direction ay·t, and the inclination in the XY plane ω·t as device movement information.

[0031] Figure 5 is a block diagram showing the schematic configuration of an information processing device 500 that performs image restoration processing using text data generated by the information processing device 100 shown in Figure 1 as a prompt. The information processing device 500 performs video image restoration processing based on the main subject image, the main subject's position information, the device movement information, and image size information generated frame by the information processing device 100 in Figure 1, as well as text data (prompts) generated according to predetermined judgment conditions. In this embodiment, the information processing device 100 in Figure 1 and the information processing device 500 in Figure 5 are shown as separate devices, but they may be a single information processing device.

[0032] The information processing device 500 has a configuration in which a CPU 501, non-volatile memory 502, memory 503, information acquisition unit 505, text determination unit 506, image generation unit 507, image restoration unit 508, etc. are connected via a system bus 504.

[0033] The CPU 501 controls the operation of each component connected via the system bus 504 by executing a program stored in the non-volatile memory 502. The non-volatile memory 502 is a non-volatile memory that can be electrically erased and saved, similar to the non-volatile memory 102 in Figure 1. The non-volatile memory 502 stores various programs for the operation of the CPU 501, and a third pre-trained model used in the image generation unit 507, which will be described later. However, this is just one example, and if neural network computation processing is not used in the image generation unit 507 or the text determination unit 506, it is not necessary to store the third pre-trained model. Also, the third pre-trained model may be a combination of one or more pre-trained models, such as a VLM (Visual Language Model) and a diffusion model. Memory 503 is a rewritable volatile memory, similar to memory 103 in Figure 1. The CPU 501 controls the operation of each component by executing a program read from non-volatile memory 502 and loaded into memory 503, and also uses memory 503 as work memory.

[0034] Furthermore, the non-volatile memory 502 may store information processing programs that enable the CPU 501 to implement the functions of the information acquisition unit 505, text determination unit 506, image generation unit 507, and image restoration unit 508, which will be described later. In this case, the CPU 501 can implement these functions by executing the information processing programs that it has read from the non-volatile memory 502 and expanded into memory 503.

[0035] The information acquisition unit 505 acquires the main subject image, the main subject's position information, the device movement information, image size information, and text data as a prompt, generated frame by the information processing device 100 in Figure 1, and stores them in the memory 503. The information acquisition unit 505 may acquire this information via a communication channel, or via a removable memory card (not shown), for example.

[0036] The text determination unit 506 determines whether or not there is text data associated with the main subject image in the memory 503. If the text determination unit 506 determines that there is text data associated with the main subject image, it instructs the image generation unit 507 to generate an image based on that text data.

[0037] The image generation unit 507 reads text data from the memory 503 and inputs this text data as a prompt to the calculation process using the third pre-trained model. The third pre-trained model is a pre-trained model that uses the input text data as a prompt and generates an image based on that prompt. In this embodiment, the information processing device 100 shown in Figure 1 generates text data representing the features of the background image, so the image generation unit 507 generates the background image by using this text data as a prompt.

[0038] Here, the image generation unit 507 of the information processing device 500 in Figure 5 generates a large-size image (hereinafter referred to as a large-size background image) that is larger in size than the image captured by the imaging unit 105 of the information processing device 100 in Figure 1, as a background image to be generated based on the prompt. Then, the image generation unit 507 extracts a background image with the same image size as the captured image from the generated large-size background image based on the device movement information, and stores the extracted background image in the memory 103.

[0039] The image restoration unit 508 reads the main subject image, the main subject's position information, and the extracted background image from the memory 503, and generates a restored image by superimposing the main subject image onto the extracted background image at the position indicated by the main subject's position information. The image restoration unit 508 then saves the restored image to the memory 503.

[0040] Figure 6 is a flowchart showing the flow of the image restoration process performed in the information processing device 100 shown in Figure 5. First, in step S600, the information acquisition unit 505 acquires the main subject image, the main subject's position information, and the device movement information generated frame by the information processing device 100 in Figure 1, as well as image size information and text data as a prompt, and stores them in the memory 503.

[0041] Next, in step S601, the text determination unit 506 determines whether or not there is text data associated with the main subject image. If it determines that there is text data associated with the main subject image, the text determination unit 506 instructs the image generation unit 507 to generate an image based on the text data. The CPU 501 then proceeds to step S602, which is performed by the image generation unit 507. On the other hand, if it determines that there is no text data associated with the main subject image, the CPU 501 proceeds to step S603, which is performed by the image generation unit 507.

[0042] In step S602, the image generation unit 507 reads text data and image size information from memory 103, and also reads the third trained model from non-volatile memory 502. The image generation unit 507 then uses the text data as a prompt to input into a calculation process that combines one or more of the third trained models, thereby generating a large-size image that is larger than the image size indicated by the image size information. In this embodiment, since the text data is a prompt representing the features of the background image, the image generation unit 507 generates a large-size background image that is larger than the image size indicated by the image size information. The image generation unit 507 then stores this large-size background image in memory 103.

[0043] When the process proceeds to step S603, the image generation unit 507 generates an extracted background image by extracting an image of the same size as the captured image from the large-size background image based on the device movement information read from the memory 103. If, in step S601, it is determined that text data exists, and in step S602, a large background image is generated and the process proceeds to step S603, the image generation unit 507 sets the center point of the large background image as the center point of the extracted background image. The image generation unit 507 then saves the extracted background image to the memory 103. On the other hand, if it is determined in step S601 that there is no text data, the image generation unit 507 uses the position based on the movement information as the center point of the extracted background image. In this case, the image generation unit 507 calculates the difference between the position based on the movement information newly acquired by the information acquisition unit 505 and the center point of the previously extracted background image, and generates the current extracted background image from the large-size background image based on that difference. The image generation unit 507 then saves the extracted background image to the memory 103.

[0044] Next, in step S604, the image restoration unit 508 reads the extracted background image from the memory 103, and then reads the main subject image and the main subject's position information from the memory 103. The image restoration unit 508 then superimposes the main subject image onto the extracted background image at the position indicated by the main subject's position information. As a result, the information processing device 500 in Figure 5 restores an image similar to the captured image 300 in Figure 3(a) acquired by the imaging unit 105 in Figure 1.

[0045] Subsequently, in step S605, the CPU 501 determines whether to terminate the process shown in the flowchart in Figure 5, based on factors such as whether or not there is a termination instruction from the user, or whether or not there is an image to be captured for the next frame. If the CPU 501 determines that it will not terminate, it returns to step S600; on the other hand, if it determines that it will terminate, it terminates the process shown in the flowchart in Figure 6.

[0046] Thus, in the information processing device 500 according to this embodiment, a background image can be generated using text data representing the features of the background image as a prompt, and by superimposing the main subject image onto the background image, an image close to the original captured image can be restored.

[0047] The process from step S606, generation of the extracted background image, to step S604, image restoration, as shown in the flowchart of Figure 6, will be explained using the image example shown in Figure 7. In this embodiment, the image generation unit 507 generates noise data by inputting a prompt to a trained VLM in step S602, and then the trained diffusion model performs inverse diffusion processing using this noise data as input to generate an image. This is a typical example of text-to-image generation.

[0048] Figure 7(a) shows an example of a large-size background image 700 generated by the image generation unit 507 using the text data 303 exemplified in Figure 3(e), and an example of the background image extraction position based on the image size indicated by the image size information and the device movement information. The large-size background image 700 is an image larger than the image size indicated by the image size information of the captured image, and the extracted background image 701 is an image with the image size indicated by the image size information.

[0049] As mentioned earlier, if text data is available, the center point of the large background image 700 becomes the center point of the extracted background image 701. On the other hand, if there is no text data, the position of the center point of the extracted background image 701 extracted from the large background image 700 will be the difference in position from the center point of the previous extracted background image, by the amount of the device movement information. In other words, in this case, the image generation unit 507 calculates the difference from the center point of the previously extracted background image, which has already been extracted, based on the device movement information newly acquired by the information acquisition unit 505, and sets the position based on that difference as the center point of the current extracted background image. For example, if the horizontal direction of the large background image 700 is the X-axis and the vertical direction is the Y-axis, the center point of the extracted background image 701 will be the position moved from the center point of the previous extracted background image by ax·t in the X-axis direction, ay·t in the Y-axis direction, and an inclination ω·t in the XY plane.

[0050] Figure 7(b) shows the extracted background image 701, which was extracted from the large-size background image 700. Figure 7(c) shows an example of a reconstructed image 702 obtained by the image reconstruction unit 508 by superimposing the main subject image from Figure 3(c) onto the extracted background image 701 based on the main subject's position information. As shown in Figure 3(c), the reconstructed image 702 is an image in which the main subject, person 301, is superimposed on the extracted background image 701 based on the main subject's position information. As shown in Figure 7(c), the reconstructed image 702 does not perfectly match the captured image 300 shown in Figure 3(a), but it is generated as an image with minimal differences.

[0051] As described above, the information processing devices 100 and 500 according to this embodiment reduce the processing load while suppressing a decrease in image quality by limiting the prompting to background images excluding the main subject.

[0052] In the embodiment described above, the generation determination unit 109 used the elapsed time since the previous text data was generated as a determination condition for determining whether or not to generate text data representing the features of the background image. However, it is not limited to this, and other determination conditions may be used. For example, the generation determination unit 109 may use the change in brightness of the background image since the previous text data was generated, i.e., the change in luminance, as a condition for determining whether to generate text data representing the features of the background image. In this case, the generation determination unit 109 calculates the average value of the luminance values ​​of the background image for each frame. Furthermore, the generation determination unit 109 obtains the difference between the calculated average value of luminance values ​​and the average value of the luminance values ​​of the background image when the previous text data was generated as the change in luminance. The generation determination unit 109 then compares this change in luminance (difference in average values ​​of luminance) with a threshold for the change in luminance stored in advance in the non-volatile memory 102, and determines that if the change in luminance exceeds the threshold for the change in luminance, it will generate text data representing the features of the background image. On the other hand, if the change in luminance is less than or equal to the threshold for the change in luminance, the generation determination unit 109 instructs the movement information generation unit 108 to generate device movement information, similar to the embodiment described above.

[0053] For example, the generation determination unit 109 may use the amount of change in the color tone of the background image after the previous text data was generated as a determination condition for generating text data representing the features of the background image. In this case, the generation determination unit 109 calculates RGB (Red, Green, Blue) values ​​as the color components of the background image for each frame. Furthermore, the generation determination unit 109 obtains a value as the amount of color change by summing the differences for each RGB color component between the calculated RGB values ​​and the RGB values ​​of the background image when the previous text data was generated. The generation determination unit 109 then compares this amount of color change with a color change threshold of RGB values ​​that has been previously stored in the non-volatile memory 102. If the amount of color change exceeds the color change threshold, the generation determination unit 109 determines to generate text data representing the features of the background image. On the other hand, if the amount of color change is less than or equal to the color change threshold, the generation determination unit 109 instructs the movement information generation unit 108 to generate device movement information, similar to the embodiment described above.

[0054] For example, the generation determination unit 109 may calculate both the amount of brightness change and the amount of color change as described above. In this case, the generation determination unit 109 may determine to generate text data representing the features of the background image when both conditions are met: the amount of brightness change exceeds the brightness change threshold, and the amount of color change exceeds the color change threshold. Furthermore, for example, the generation determination unit 109 may determine to generate text data when at least two conditions are met: the elapsed time exceeds the time threshold, the amount of brightness change exceeds the brightness change threshold, and the amount of color change exceeds the color change threshold. As mentioned above, the generation determination unit 109 can determine whether or not to generate text data representing the features of the background image, even under conditions other than elapsed time.

[0055] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more of the functions. The embodiments described above are merely examples of how the present invention can be implemented, and the technical scope of the present invention should not be interpreted as being limited by them. In other words, the present invention can be implemented in various ways without departing from its technical concept or its main features.

[0056] This embodiment includes the following configurations, methods, and programs. (Composition 1) A detection means that detects a predetermined object from an input image and generates an image of the object and a background image, A text generation means for generating text data representing the features of the background image, It has, An information processing apparatus characterized by storing or transmitting an image of the object detected by the detection means and text data generated by the text generation means. (Configuration 2) The input image is an captured image, and the predetermined object is a predetermined subject that appears in the captured image. The system further includes a movement information generation means that generates movement information indicating the direction and amount of movement of the imaging unit that acquired the captured image, The detection means also acquires the position information of the subject in the captured image and the image size information of the captured image. The information processing device according to Configuration 1, characterized in that, in addition to the image of the subject detected by the detection means and the text data generated by the text generation means, the location information and image size information of the subject and the movement information generated by the movement information generation means are stored or transmitted. (Composition 3) The information processing apparatus according to configuration 2, characterized in that the movement information generation means generates the movement information based on the acceleration and angular velocity of the imaging unit that acquires the captured image. (Composition 4) The system further includes a generation determination means that determines whether or not to generate the text data based on predetermined determination conditions, The generation determination means is If it is determined that the aforementioned text data should be generated, the text generation means is instructed to generate text data representing the features of the background image. The information processing device according to configuration 2 or 3, characterized in that, if it is determined that the text data is not to be generated, the device instructs the movement information generation means to generate the movement information. (Composition 5) The information processing apparatus according to configuration 4, wherein the generation determination means uses the elapsed time after determining that text data should be generated as a predetermined determination condition, and determines that new text data should be generated when the elapsed time exceeds a predetermined time threshold. (Composition 6) The information processing device according to configuration 4, wherein the generation determination means uses the amount of change in the brightness value of the background image after it has been determined that text data should be generated as a predetermined determination condition, and determines that text data should be generated when the amount of change in the brightness value exceeds a predetermined change threshold. (Composition 7) The information processing apparatus according to configuration 4, wherein the generation determination means uses the amount of change in the color of the background image after it has been determined that text data should be generated as a predetermined determination condition, and determines that text data should be generated when the amount of change in color exceeds a predetermined change threshold. (Composition 8) The information processing apparatus according to any one of configurations 1 to 7, characterized in that the text generation means generates the text data according to the set level of detail. (Composition 9 places) Information acquisition means for acquiring an image of a predetermined object and text data representing the characteristics of the background image, Image generation means for generating a background image based on text data representing the characteristics of the aforementioned background image, Image restoration means for superimposing an image of a predetermined object onto a background image generated by the image generation means, An information processing device characterized by having the following features. (Composition 10) The image of the predetermined object is an image of the subject extracted from the captured image. The information acquisition means also acquires the position information of the subject in the captured image and the image size information of the captured image. The image generation means generates a large image larger than the image size information based on the text data, and extracts the background image of the image size information from the large image. The information processing apparatus according to configuration 9, wherein the image restoration means superimposes an image of the subject on the background image at a position indicated by the subject's position information. (Composition 11) The information acquisition means also acquires movement information indicating the direction and amount of movement of the imaging unit that acquired the captured image. The information processing apparatus according to configuration 10, characterized in that the image generation means extracts the background image of the image size information from the large-size image based on the movement information. (Composition 12) The image generation means is When the information acquisition means acquires the text data, the center point of the large-size image is set as the center point of the background image extracted using the image size information. The information processing apparatus according to configuration 11, characterized in that, if the information acquisition means is unable to acquire the text data, the position based on the movement information is set to the center point of the background image extracted with the image size information. (Composition 13) The information processing apparatus according to configuration 12, characterized in that the image generation means calculates the difference between the position based on the movement information newly acquired by the information acquisition means and the center point of the previously extracted background image, and extracts the background image of the image size information based on the difference. (Method 1) A detection step that detects a predetermined object from an input image and generates an image of the object and a background image, A text generation step that generates text data representing the features of the background image, It has, An information processing method characterized by saving or transmitting an image of the object detected in the detection step and text data generated in the text generation step. (Method 2) An information acquisition process that acquires an image of a predetermined object and text data representing the characteristics of the background image, An image generation process that generates a background image based on text data representing the characteristics of the aforementioned background image, An image restoration step is performed in which an image of a predetermined object is superimposed on the background image generated by the image generation step, An information processing method characterized by having the following features. (Program 1) A program that causes a computer to function as an information processing device described in any one of configurations 1 to 13. [Explanation of Symbols]

[0057] 100, 500: Information processing device, 106: Subject detection unit, 107: Text generation unit, 108: Movement information generation unit, 109: Generation determination unit, 505: Information acquisition unit, 506: Text determination unit, 507: Image generation unit, 508: Image restoration unit

Claims

1. A detection means that detects a predetermined object from an input image and generates an image of the object and a background image, A text generation means for generating text data representing the features of the background image, It has, An information processing apparatus characterized by storing or transmitting an image of the object detected by the detection means and text data generated by the text generation means.

2. The input image is an captured image, and the predetermined object is a predetermined subject that appears in the captured image. The system further includes a movement information generation means that generates movement information indicating the direction and amount of movement of the imaging unit that acquired the captured image, The detection means also acquires the position information of the subject in the captured image and the image size information of the captured image. The information processing apparatus according to claim 1, characterized in that, in addition to the image of the subject detected by the detection means and the text data generated by the text generation means, the apparatus also stores or transmits the location information and image size information of the subject and the movement information generated by the movement information generation means.

3. The information processing apparatus according to claim 2, characterized in that the movement information generation means generates the movement information based on the acceleration and angular velocity of the imaging unit that acquires the captured image.

4. The system further includes a generation determination means that determines whether or not to generate the text data based on predetermined determination conditions, The generation determination means is If it is determined that the aforementioned text data should be generated, the text generation means is instructed to generate text data representing the features of the background image. The information processing device according to claim 2, characterized in that, if it is determined that the text data is not to be generated, the device instructs the movement information generation means to generate the movement information.

5. The information processing apparatus according to claim 4, characterized in that the generation determination means uses the elapsed time after it has been determined that text data should be generated as a predetermined determination condition, and determines that new text data should be generated when the elapsed time exceeds a predetermined time threshold.

6. The information processing apparatus according to claim 4, characterized in that the generation determination means uses the amount of change in the brightness value of the background image after it has been determined that text data should be generated as a predetermined determination condition, and determines that text data should be generated when the amount of change in the brightness value exceeds a predetermined change threshold.

7. The information processing apparatus according to claim 4, characterized in that the generation determination means uses the amount of change in the color of the background image after it has been determined that text data should be generated as a predetermined determination condition, and determines that text data should be generated when the amount of change in color exceeds a predetermined change threshold.

8. The information processing apparatus according to claim 1, characterized in that the text generation means generates the text data according to the set level of detail.

9. Information acquisition means for acquiring an image of a predetermined object and text data representing the characteristics of the background image, Image generation means for generating a background image based on text data representing the characteristics of the aforementioned background image, Image restoration means for superimposing an image of a predetermined object onto a background image generated by the image generation means, An information processing device characterized by having the following features.

10. The image of the predetermined object is an image of the subject extracted from the captured image. The information acquisition means also acquires the position information of the subject in the captured image and the image size information of the captured image. The image generation means generates a large image larger than the image size information based on the text data, and extracts the background image of the image size information from the large image. The information processing apparatus according to claim 9, wherein the image restoration means superimposes an image of the subject on the background image at a position indicated by the subject's position information.

11. The information acquisition means also acquires movement information indicating the direction and amount of movement of the imaging unit that acquired the captured image. The information processing apparatus according to claim 10, wherein the image generation means extracts the background image of the image size information from the large-size image based on the movement information.

12. The image generation means is When the information acquisition means acquires the text data, the center point of the large-size image is set as the center point of the background image extracted using the image size information. The information processing apparatus according to claim 11, characterized in that, if the information acquisition means is unable to acquire the text data, the position based on the movement information is set to the center point of the background image extracted with the image size information.

13. The information processing apparatus according to claim 12, characterized in that the image generation means calculates the difference between the position based on the movement information newly acquired by the information acquisition means and the center point of the previously extracted background image, and extracts the background image of the image size information based on the difference.

14. A detection step that detects a predetermined object from an input image and generates an image of the object and a background image, A text generation step that generates text data representing the features of the background image, It has, An information processing method characterized by saving or transmitting an image of the object detected in the detection step and text data generated in the text generation step.

15. An information acquisition process that acquires an image of a predetermined object and text data representing the characteristics of the background image, An image generation process that generates a background image based on text data representing the characteristics of the aforementioned background image, An image restoration step is performed in which an image of a predetermined object is superimposed on the background image generated by the image generation step, An information processing method characterized by having the following features.

16. Computers, A detection means that detects a predetermined object from an input image and generates an image of the object and a background image, A text generation means for generating text data representing the features of the background image, It has, A program that causes an information processing device to function as one that stores or transmits an image of the object detected by the detection means and text data generated by the text generation means.

17. Computers, Information acquisition means for acquiring an image of a predetermined object and text data representing the characteristics of the background image, Image generation means for generating a background image based on text data representing the characteristics of the aforementioned background image, Image restoration means for superimposing an image of a predetermined object onto a background image generated by the image generation means, A program that makes an information processing device function as having a certain feature.

Images