Information processing device, information processing system, and information processing method

Abstract

This system provides an information processing device that prioritizes photographing subjects and scenes that users are likely to purchase, based on their individual usage information for the photo service. [Solution] The image processing device includes a recognition means for recognizing a subject from captured image data, an evaluation value determination means for determining an evaluation value based on the recognition result of the subject recognized by the recognition means, a priority determination means for determining the priority of each subject according to the evaluation value determined by the evaluation value determination means, and a usage information acquisition means for acquiring usage information for each subject recognized by the recognition means, and the evaluation value determination means updates the evaluation value for each subject according to the usage information.

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus, an information processing system, and an information processing method.

Background Art

[0002] In a school photo service that photographs and sells the images of children in kindergartens and nurseries, there is a desire to eliminate the bias in the number of photographs taken for each target child and to take the best-looking photos possible. Therefore, technologies for recognizing the photographed subjects and equalizing the amount of photography for each subject, and technologies for automatically selecting good-looking photos have been proposed. For example, in Patent Document 1, it is possible to photograph a subject according to the priority based on the image quality score for each subject. Also, in Patent Document 2, it is possible to eliminate the bias in the number of photographs taken for each photographing scene.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the conventional technologies disclosed in the above-mentioned patent documents, the photography does not conform to the purchase intention of the users of the school photo service who purchase the photographed photos. For example, in Patent Document 1, it is possible to calculate the image quality score for each subject and assign a priority, but it is not possible to assign an optimized priority for each subject based on the usage history of the school photo service or the like. Also, in Patent Document 2, it is possible to eliminate the bias in the number of photographs taken for each photographing scene, but it is not possible to photograph with the aim of taking photos of the scenes that the users of the school photo service want to purchase.

[0005] The present invention was made to solve the above-mentioned problems, and aims to provide an information processing device that prioritizes capturing scenes that are likely to be purchased by each user of a photo service, based on the user's usage information. [Means for solving the problem]

[0006] As one means to achieve the above objective, the information processing device of the present invention comprises: recognition means for recognizing a subject from captured image data; evaluation value determination means for determining an evaluation value based on the recognition result of the subject recognized by the recognition means; priority determination means for determining the priority of each subject according to the evaluation value in the evaluation value determination means; and usage information acquisition means for acquiring usage information for each subject recognized by the recognition means, wherein the evaluation value determination means updates the evaluation value for each subject according to the usage information. [Effects of the Invention]

[0007] According to the present invention, it is possible to prioritize capturing images that are likely to be used by the user. [Brief explanation of the drawing]

[0008] [Figure 1] This is a block diagram of the hardware of Example 1. [Figure 2] This is a functional block diagram of Example 1. [Figure 3] This is a diagram illustrating the captioning method used in Example 1. [Figure 4] This is a flowchart of the process for automatically photographing the target subject until the target amount of images to be captured in Example 1 is reached. [Figure 5] This is a flowchart for calculating evaluation values ​​for multiple subjects in Example 1. [Figure 6] This figure illustrates an example of the image capture control in Example 1. [Figure 7] This figure illustrates an example of the image quality score and scene recognition data from Example 1. [Figure 8]This is an example showing the subject information data for Example 1. [Figure 9] This is an example of a lookup table for determining the usage performance index in Example 1. [Figure 10] This flowchart shows an example of saving the video clip from Example 2. [Figure 11] This flowchart shows an example of the processing flow for shooting a video clip in Example 2. [Figure 12(A)] This is a processing flow illustrating an example of video clip creation in Example 3. [Figure 12(B)] This is a processing flow showing an example of determining the evaluation value for each frame in Figure 12(A). [Figure 13] This is an illustrative diagram of the video clip shooting process in Example 3. [Modes for carrying out the invention]

[0009] The present invention will now be described in detail based on its preferred embodiments with reference to the attached drawings. Note that the configurations shown in the following embodiments are merely examples, and the present invention is not limited to the illustrated configurations. <Embodiment 1>

[0010] In this embodiment, we will explain an example of the application of the present invention in a school photo service that uses a network camera to automatically take pictures of kindergarten children and sells those pictures on the web.

[0011] Figure 1 shows an example of the hardware configuration of the information processing device 100 according to this embodiment. The CPU 101 is a CPU (Central Processing Unit) that performs calculations and logical decisions for various processes and controls each component connected to the system bus 111.

[0012] This information processing apparatus 100 is equipped with a memory including a program memory and a data memory. A ROM (Read Only Memory) 102 is a program memory that stores a program for control by a CPU including various processing procedures described later.

[0013] A RAM (Random Access Memory) 103 is a data memory that has a work area for the above program of the CPU 101, a data backup area during error processing, a load area for the above control program, and the like. Note that a program memory may be realized by loading a program from an external storage device or the like connected to the information processing apparatus 100 into the RAM 103.

[0014] An auxiliary storage device 104 is an auxiliary storage device for storing a plurality of electronic data and programs according to the present embodiment. An external storage device may be used to perform a similar role. Here, the external storage device can be realized, for example, by a medium (recording medium) and an external storage drive for realizing access to the medium. As such a medium, for example, a flexible disk (FD), a CD-ROM, a DVD, a USB memory, a MO, a flash memory, etc. are known. Further, the external storage device may be a server device or the like connected by a network.

[0015] An input device 105 is a device for taking in the user's operation information into the information processing apparatus 100. For example, it includes pointing devices such as a mouse and a touch panel, a joystick, a keyboard, etc. An output device 106 is a device for performing display output, and for example, includes a monitor and a printer.

[0016] A communication interface (I / F) 107 performs bidirectional communication, either wired or wireless, with other information processing apparatuses, communication devices, external storage devices, etc. according to a known communication technology. In the present embodiment, the communication I / F 107 is connected to an imaging device 110 which is a network camera via a LAN, and receives the captured video in real time.

[0017] The imaging device 110 is a network camera that is installed connected to a network and captures images of the faces of people entering the room within its field of view. The imaging device 110 is not limited to a network camera; a webcam integrated into this information processing device may also be used. Other types of cameras, such as USB-connected cameras, may also be used.

[0018] The Web service DB 108 is a database that records usage data, including the purchase history of each user of the photo service. The information processing device 100 can access the Web service DB 108 via the network 109.

[0019] Figure 2 is a block diagram showing an example of the functional configuration of the information processing device 100. The information processing device 100 includes a priority determination unit 112, an evaluation value determination unit 113, an information management unit 114, an image capture control unit 115, a recognition unit 116, a usage information acquisition unit 117, an image capture unit 118, a recording unit 119, and an assignment unit 120.

[0020] Each of these functional units is realized when the CPU 101 loads the program stored in the ROM 102 into the RAM 103 and executes the processing according to the flowcharts described later.

[0021] The recording unit 119 is a functional unit of the RAM 103. However, embodiments of the present invention can also be implemented using an information processing device that implements these functional units in hardware.

[0022] The following describes each element. The priority determination unit 112 determines the priority for each subject. The priority is determined based on the evaluation value for each subject determined by the evaluation value determination unit 113.

[0023] The evaluation value determination unit 113 determines an evaluation value for each subject based on the captured image data taken by the imaging device 110. Here, captured image data refers to either still image data or video data. Specifically, the evaluation value for each subject is determined based on a score representing the degree to which the subject's face is captured and a variety score that quantifies the diversity of the shooting scene. Furthermore, by optimizing the score calculation method for each subject, the preferred shooting situation for each subject can be optimized. Here, the shooting scene includes at least one piece of information such as other subjects in the image, location, subject's actions, and time of day of shooting.

[0024] To optimize the method for determining evaluation values ​​for each subject, the information management unit 114 registers the subjects to be photographed. The usage information acquisition unit 117 acquires usage information for each user based on the usage history of the Web service.

[0025] Furthermore, the information management unit 114 decides whether to continue or end shooting for each subject based on the cumulative evaluation value obtained by accumulating evaluation values ​​for each subject in images that have already been taken. Through these processes, it becomes possible to prioritize shooting scenes that are likely to be purchased by users of the school photo service and to reduce bias in the shooting scenes for each subject.

[0026] The information management unit 114 manages the usage information for each subject acquired by the usage information acquisition unit 117, as well as the cumulative evaluation value for each subject for images already taken. Based on this subject information, it takes pictures of each subject evenly. For example, the information management unit 114 may prioritize taking pictures of subjects with low cumulative evaluation values, or take more pictures of subjects with a high number of past purchases or views.

[0027] Furthermore, the information management unit 114 registers the subjects to be photographed and readjusts the cumulative evaluation value according to the increase or decrease in the number of subjects to be photographed. Based on the readjusted cumulative evaluation value, photography is carried out evenly.

[0028] The shooting control unit 115 performs shooting control processing according to the priority determined by the priority determination unit 112. Specifically, it performs a cropping process on the image acquired by the imaging device 110 to extract a region of an arbitrary aspect ratio that contains a high-priority subject. In other words, for each subject recognized by the recognition means, the shooting control unit 115 creates an image by extracting a portion of the captured image data based on the evaluation value.

[0029] Furthermore, the CPU 101 issues control signals to the imaging device 110, which is a network camera connected via the communication IF 107, and controls the pan, tilt, and zoom operations of the imaging device 110 to match the movement of the subject.

[0030] Furthermore, the shooting control unit 115 may perform optimal camera control based on the subject information and shooting scene information recognized by the recognition unit 116, which will be described later. For example, if the subject's face is prominently displayed and the evaluation value for facial expression is high, the shooting control unit 115 switches the camera's shooting mode to portrait mode. Also, in scenes with a lot of movement, the shooting control unit 115 switches to shutter speed priority mode, making it possible to take high-quality photos that better match the shooting scene. In addition, for example, the shooting control unit 115 switches between multiple cameras, or if the camera is movable, moves it to switch the scene being shot.

[0031] The recognition unit 116 uses recognition technology to detect the subject, recognize the subject's face, and recognize the shooting scene on the captured image data acquired by the shooting unit 118. Here, the captured image data is data obtained from a shooting device that photographs the subject. For face recognition, any known image recognition technology can be used. The facial features obtained by inputting the face image into a known image recognition model pre-trained with a machine learning algorithm are compared with the facial features of children that have been registered in advance to determine whether it is the correct person.

[0032] Furthermore, scene recognition by the recognition unit 116 can be performed using existing known methods. For example, it can be achieved by performing object detection on an image, estimating "what" is depicted and "where" it is, and then estimating the scene from the positional relationships of the detected objects. Alternatively, action recognition can be performed to estimate "what" the subject is doing. In recent years, many scene recognition and action recognition technologies using known image recognition models have been researched and developed, so it can be achieved by using these. In this embodiment, we will use scene recognition and action recognition technologies using known image recognition models.

[0033] The user information acquisition unit 117 accesses the Web service DB 108 via the communication IF 107 and acquires user information, which includes purchase history information and browsing history information for each user of the school photo service.

[0034] The imaging unit 118 acquires image data or video data captured by the imaging device 110.

[0035] The recording unit 119 temporarily stores the captured image data or captured video data acquired by the shooting unit 118, or stores it as electronic data in another storage device.

[0036] The captioning unit 120 adds caption information to the images to be saved by the recording unit 119 when the images acquired by the shooting unit 118 are saved, based on the results recognized by the recognition unit 116. Specifically, it generates and adds descriptive text to the image based on the information of the subject and the shooting scene recognized by the recognition unit 116. For adding caption information, any known captioning technology can be used. For example, there is a known technology that uses a trained AI model to automatically generate captions that describe the content of an image, and this can be used to achieve this.

[0037] Figure 3 shows image 500 with captions. Captions may be created for each subject in the image. For example, in a scene where child A and child B are playing with building blocks, the image distributed to child A's parents could have a caption such as "Playing with building blocks with B" inserted into the image. Captions may also include the time of shooting. For example, there may be times when you want to check how the children are doing during a specific scene, such as lunchtime or nap time. By inserting information about the start and end times into the captions of such scenes, you can understand how the children are doing at the nursery school, which will further increase the added value for the user. The above describes the configuration of the information processing device 100.

[0038] Figure 4 shows a flowchart of the process performed by the information processing device 100, specifically the process of automatically photographing the target subject until the target number of images is reached. Each step is explained below.

[0039] In S200, the imaging unit 118 acquires the image captured by the imaging device 110.

[0040] In S201, the recognition unit 116 recognizes the subject from the image acquired in S200. The recognition unit 116 uses facial recognition technology to determine which child is the subject in the image. Multiple subjects may be present in the image.

[0041] In S202, the evaluation value determination unit 113 acquires subject information from the information management unit 114 for each subject recognized by the recognition unit 116 in S201. The subject information consists of usage information for each subject acquired by the usage information acquisition unit 117 and the cumulative evaluation value for each subject for images already taken.

[0042] In S203, the evaluation value determination unit 113 updates the evaluation value for each subject recognized in S201, according to the usage information for each subject acquired in S202. Updating the evaluation values ​​for multiple subjects is processed according to the flow shown in Figure 5, which will be described later.

[0043] In S204, the priority determination unit 112 determines the priority for each subject based on the evaluation value determined in S203.

[0044] In S205, the shooting control unit 115 performs shooting control processing based on the priority of each subject determined in S204. Details of the shooting control will be described later with reference to Figure 6.

[0045] In S206, the captioning unit 120 adds a caption to the captured image based on the subject and shooting scene information recognized by the recognition unit 116.

[0046] In S207, the recording unit 119 temporarily saves the captured image data to the auxiliary storage device 104 or the like. At this time, the recording unit 119 sets a threshold th for the evaluation value and saves the image data only when the evaluation value is equal to or greater than the threshold th. The recording unit 119 may set this threshold th to a fixed value in advance, or the system user may change the threshold during the process. Alternatively, the recording unit 119 may set the threshold th low at the start of shooting, automatically change the threshold based on the number of shots for each subject, increase the number of shots at the start of shooting, and then prioritize shooting subjects with high evaluation values ​​as the number of shots increases.

[0047] In S208, the information management unit 114 updates the cumulative evaluation value for each subject being managed.

[0048] In S209, the information management unit 114 determines whether the cumulative evaluation value for each managed subject has reached the target. If the cumulative evaluation value has not reached the target, the shooting control unit 115 returns to S200 and continues automatic shooting; if the target amount of shooting has been reached, it terminates automatic shooting. The shooting control unit 115 continues to shoot each subject until the cumulative evaluation value for each subject, according to the usage information, reaches a predetermined value (cumulative target value).

[0049] At S209, the shooting control unit 115 may continue shooting even if the target number of images has been reached. In that case, the information management unit 114 raises the threshold value for the evaluation value when saving image data at S207 and replaces the previously saved images with newly captured images. At that time, the information management unit 114 replaces the images with newly captured images in order from the images with the lowest evaluation values, so that more images with higher evaluation values ​​can be saved even with the same number of shots. The flow of the automatic shooting process shown in Figure 4 has been explained above.

[0050] Next, using the flowchart in Figure 5, we will explain the process by which the evaluation value determination unit 113 determines an evaluation value for each subject in step S203 of Figure 4.

[0051] In S300 of Figure 5, the evaluation value determination unit 113 processes each subject in the captured image data taken by the imaging device 110.

[0052] In S301, the evaluation value determination unit 113 determines the score Q for the quality of the subject's face. Various methods, such as those disclosed below, can be used to determine the score Q for the quality of the face. Schlett, Torsten, et al. “Face image quality assessment:A literature survey.” ACM Computing Surveys(CSUR) 54.10s(2022):1-49.

[0053] The image quality score Q is a scalar value ranging from 0.0 to 1.0, based on one or more factors related to the image quality of the face. These factors include, for example, the brightness distribution related to the image quality of the subject, the degree of backlighting or oblique lighting, the degree of overexposure or underexposure, the degree of dynamic range, and the degree of blur or out-of-focus. Furthermore, the subject's facial expression, face size, face orientation, degree of eye and mouth opening / closing, and degree of facial occlusion are also included as factors.

[0054] The function fq for calculating the image quality score Q can be obtained as follows using the above example of factors.

[0055] Q = fq (luminance distribution, degree of backlighting or oblique lighting, degree of overexposure / underexposure, degree of dynamic range, degree of blur or out-of-focus, face size, face orientation, facial expression, degree of eye and mouth opening / closing, degree of face occlusion)

[0056] The function fq above is expressed as a mathematical formula with each factor as a variable.

[0057] For example, the coefficients of the function fq can be set so that the image quality score Q is high if the face is facing forward, there is no blurring or out-of-focus, and the person is smiling. Conversely, even if the above conditions for a high score are met, the function fq may be set so that the score is low if the eyes are closed or there is an obstruction in front of the face.

[0058] Alternatively, for example, based on the information obtained by the usage information acquisition unit 117, a decision method could be used that reflects the importance of each factor from the facial image quality score of photos purchased through the school photo service. In this way, the coefficients for each variable in the function fq can be adjusted manually or determined by machine learning. Furthermore, the number of purchases and views by users may be used when determining each coefficient.

[0059] In S302, the recognition unit 116 recognizes the shooting scene from the captured image data captured by the imaging device 110. The recognition by the recognition unit 116 can be performed using known methods. For example, there is a known technique that performs object detection on an image, estimates "what" is in the image and "where" it is, and then estimates the scene and actions from the positional relationship of the detected objects, and this can be used to achieve the recognition.

[0060] Alternatively, the recognition unit 116 may perform action recognition to estimate "what the subject is doing." Alternatively, scene recognition or action recognition techniques using known image recognition models may be used. In this embodiment, scene recognition and action recognition techniques using known image recognition models will be used.

[0061] Here, an example of the data format representing the image quality score for each subject and the scene recognition results in S302 is shown in Figure 7, T100.

[0062] Based on the image data captured by the imaging device 110, the image quality score determined by the recognition unit 116 for each subject and the scene recognition results are recorded in the auxiliary storage device 104. The person ID and location information of the child obtained as a result of the subject recognition S201 by the recognition unit 116, the image quality score determined in S301 for the face image, and the shooting scene recognition results recognized in S302 are recorded in the auxiliary storage device 104. The shooting scene recognition results may be recorded separately as information about the shooting location, information about the subject, and information about the subject's actions. In this embodiment, the shooting scene number corresponding to the combination of these scene recognition results is represented by a sequential number.

[0063] In Figure 5, at S304, weighting is performed on the results determined in S301 and the shooting scene recognition results determined in S302, based on the subject information management unit 114 which manages information for each subject.

[0064] The information for each subject, managed by the information management unit 114 and used for weighting in S304, is explained in Figure 8, T200.

[0065] Figure 8 shows the current number of photos taken, the target number of photos taken, the average image quality score, and the scene variety score representing the diversity of the shooting scenes for each individual ID of the target child.

[0066] Furthermore, Figure 8 displays information on the score derived from the purchase history of each person ID, as well as preferred shooting scenes, obtained from accounts associated with person IDs in the school photo service.

[0067] Based on this subject information, a weighting process is performed on the image quality score determined in S301 and the recognition result of the shooting scene determined in S302, and the result of the evaluation value determination process in S203 is determined as the evaluation value V. The evaluation value V can be defined by the following formula.

[0068] V = Image quality score × Weight W

[0069] The weight W can be calculated using the following function fw.

[0070] W = fw (number of shots, target number of shots, usage index K, average image quality score, scene variety score)

[0071] The utilization index K can be calculated using the following function fk.

[0072] K = fk (purchase history, number of views)

[0073] Here, the usage index K is a numerical representation of the user's motivation to purchase photos, reflecting their preferences. This can be obtained by analyzing web service usage history information, such as purchase history and browsing history, of user accounts in the school photo service.

[0074] A high value for the usage index K indicates that users have a high willingness to purchase, suggesting a high probability of them making a purchase.

[0075] For example, one could use the sum of the values ​​obtained by adding up each parameter, as shown in the formula below.

[0076] K = Number of purchases + Number of views

[0077] Furthermore, if the range of possible values ​​for the exponent K in the above formula is too wide, you may limit the range of possible values ​​for each parameter and K, and modify the function or convert the values ​​using a lookup table so that they fit within the range.

[0078] An example of a lookup table is shown in Figure 9, T300. T300 shows the value of the usage index K when the horizontal axis is the purchase history and the vertical axis is the number of views.

[0079] Here, usage history information such as the number of purchases and the number of views is obtained from log information stored in Web service DB108. The number of purchases is determined from the number of photos, etc., purchased by the user within a certain period. The number of views is determined from the number of times the user viewed photos, etc., from a web page within a certain period.

[0080] By using the usage index K, for example, based on the purchase history score, subjects associated with accounts with a high number of purchases are given greater weight, increasing their evaluation value and making it easier for the captured photos to be saved to the auxiliary storage device 104.

[0081] Alternatively, a weight W may be calculated for each scene. For example, if there are few shooting scenes in the captured data, the weight W may be increased, while if there is already a good amount of data from various scenes and the scene variety score is high, the weight W may be decreased. By performing this process, the evaluation value can be adjusted for each subject. The above explains the process for determining the evaluation value for each subject, as shown in Figure 5.

[0082] Next, the process of image capture control in step S205 in Figure 4 will be explained using Figure 6. The image capture control unit 115 in Figure 2 performs processing based on the priority determined in the previous step S204.

[0083] Various processing methods are possible. For example, as shown in Figure 6, the shooting control unit 115 processes the captured image 400 to crop the region 402 containing the subject 401 with the highest priority, using an arbitrary aspect ratio. The rules for this cropping can be predetermined, for example.

[0084] Alternatively, the cropped image may be cropped so that it includes as many other subjects as possible, or the area occupied by the face of the preferred subject may be determined. In this embodiment, the image is cropped in a vertical aspect ratio intended for viewing on smartphones, making it easier for users of the school photo service to view, and is expected to increase the frequency of access to the service.

[0085] For example, the CPU 101 issues pan, tilt, and zoom operation commands to the network camera, which is an imaging device 110 connected via the communication IF 107 and network 109, in accordance with the movement of the priority subject. This makes it possible to obtain images that focus more on the priority subject and capture images with a higher evaluation value.

[0086] For example, the shooting control unit 115 instructs the imaging device 110 to switch shooting modes according to the priority subject. This makes it possible to shoot in the optimal shooting mode according to the movement and state of the subject. The shooting control process shown in Figure 6 has been explained above.

[0087] As explained above, in Embodiment 1, the school photo service automatically takes photos while suppressing bias in the number of shots and shooting scenes of subjects, and estimates the preferred scenes for each subject based on usage information. This allows the user's preferences to be reflected, making it possible to create photos and videos that users are more likely to purchase. <Modified form of Embodiment 1>

[0088] In S207 of Figure 4, the image data captured does not necessarily have to be saved to the auxiliary storage device 104. Alternatively, the captured data can be saved by connecting to the Web service DB 108 via the network 109. This allows for direct updating of the content on the server providing the Web service, shortening the time from capture to service provision, and enabling more timely delivery of captured data to users. <Embodiment 2>

[0089] Embodiment 1 described an example of selling photos taken using a network camera through a school photo service, but this can also be applied to selling video clips. An embodiment for video clips is described below. Note that the same parts as in Embodiment 1 will be omitted from the explanation.

[0090] Figure 10 shows the process flow for saving as a video clip. The save process at S207 in the flowchart shown in Figure 4 has been removed, and the shooting control at S205 has been changed to the video clip creation process at S600. The processes other than S600 are the same as those in Figure 4, so their explanation is omitted.

[0091] Figure 11 is a flowchart of the video clip creation process. Hereafter, each step will be described by adding the letter S to the beginning of its symbol.

[0092] In S601 of Figure 11, the evaluation value determination unit 113 determines whether the evaluation value of the priority subject determined by the priority determination in S204 is equal to or greater than the video clip start threshold th_ms.

[0093] The evaluation value determination unit 113 terminates the video clip creation process if the evaluation value of the priority subject is less than the threshold. If the evaluation value of the priority subject determined by the evaluation value determination unit 113 is equal to or greater than the threshold th_ms, the process proceeds to the video clip creation process from S602 onwards.

[0094] In S602, the shooting control unit 115 starts saving the video as a video clip.

[0095] In S603, the shooting control unit 115 acquires the image for the next frame.

[0096] In S604, the recognition unit 116 uses facial recognition technology to determine which child is depicted in the image acquired in S603.

[0097] In S605, the evaluation value determination unit 113 determines the evaluation value of the priority subject being photographed.

[0098] In S606, if the shooting control unit 115 is equal to or greater than the video clip end threshold th_me, it returns to S603, and the evaluation value determination unit 113 repeats the process of acquiring the next frame and determining the evaluation value.

[0099] The shooting control unit 115 saves the video clip and terminates processing when the evaluation value falls below the threshold th_me. At this time, the video clip termination threshold th_me is set to a different threshold than the video clip start threshold th_ms. For example, by setting a threshold smaller than the video clip start threshold th_ms, the video clip is not unnecessarily shortened even if the evaluation value temporarily decreases.

[0100] The process flow for shooting video clips, as shown in Figures 10 and 11, has been explained above. It should be noted that, in combination with Embodiment 1, still images and videos may be saved simultaneously. Doing so will increase the variety of captured content. <Embodiment 3>

[0101] Embodiments 1 and 2 describe examples in which image data captured by the imaging device 110 is processed in real time. However, video clip extraction processing may also be performed later on pre-recorded video data. This allows for separating the timing of shooting and image extraction processing for each subject. This can be used even when childcare workers shoot with handheld cameras during events such as walks or field trips at the nursery school, enabling image collection in a wide range of scenes. The detailed processing flow in this case is explained in Figure 12(A).

[0102] In the S800, the evaluation value determination unit 113 reads recorded video data of the subject as captured image data. Captured image data is data obtained from recorded video data of the subject.

[0103] In S801, the evaluation value determination unit 113 performs a process to determine an evaluation value for each frame of the recorded video data.

[0104] First, we will explain the frame-by-frame evaluation value determination process in S801 using Figure 12(B).

[0105] In S807, the evaluation value determination unit 113 performs evaluation value determination processing for each frame of the input video. In this evaluation value determination processing, the evaluation value determination unit 113 repeats the process until the evaluation value determination processing is completed for all frames of the input video.

[0106] In S808, the recognition unit 116 uses facial recognition technology to determine which child is the subject shown in the frame of the acquired video. Multiple subjects may be shown in the image.

[0107] In S809, the evaluation value determination unit 113 acquires subject information for each subject recognized by the recognition unit 116 in S808. This subject information includes the usage information for each subject acquired by the usage information acquisition unit 117, and the cumulative evaluation value for each subject for images already taken.

[0108] In S810, the evaluation value determination unit 113 determines an evaluation value for each subject recognized in S808 based on the subject information acquired in S809. For multiple subjects, the evaluation value determination is processed according to the flow described in Figure 5. Now, let's return to the explanation of the flow in Figure 12(A).

[0109] In S802, the video clip creation process is performed. The video clip creation process is the same as the process in the flows shown in Figures 10 and 11 described above. Alternatively, the evaluation value determination unit 113 may perform evaluation value determination processing for all frames of the video in advance, and then the shooting control unit 115 may selectively extract only the parts that are better suited as video clips. Specifically, the shooting control unit 115 determines the region to be extracted as a video clip, as well as the start and end frames for extraction, based on the evaluation value information for each frame saved in S811. The region to be extracted as a video clip is determined by the method described in Figure 6. Figure 13 shows how to determine the range in the time axis direction of the frames to be extracted in the region determined by the above method.

[0110] Figure 13 shows the video 701 and the fluctuation information 702 of the evaluation value for each frame in the region to be extracted on the time axis 700. As shown in Figure 13, the shooting control unit 115 extracts the range with high evaluation values ​​as a video clip based on the change in the evaluation value of the subject in the region to be extracted. Alternatively, the extraction method may involve extracting a predetermined number of frames before and after the frame with the highest evaluation value within the time period in which the evaluation value threshold was exceeded.

[0111] Furthermore, the shooting control unit 115 may extract all portions where the evaluation value exceeds a threshold. If the length (number of frames) of the extracted video clip is too long, the shooting control unit 115 may extract it to an arbitrary length or extract a random range so that it matches a predetermined number of video clips.

[0112] Furthermore, if the shooting control unit 115 determines that the time during which the evaluation value exceeds the threshold is too short to extract as a video clip, it saves the frame with the highest evaluation value as a still image.

[0113] For example, if the shooting control unit 115 exceeds the threshold in multiple time periods, it can create a video clip containing multiple scenes by extracting video clips from those time periods using the method described above and then joining those video clips together.

[0114] Furthermore, the shooting control unit 115 may create multiple video clips by changing the subject to be extracted from the same video.

[0115] In S803, the captioning unit 120 adds captions to the captured video clips based on the subject and shooting scene information recognized by the recognition unit 116.

[0116] In S804, the shooting control unit 115 saves the captured video clip to the auxiliary storage device 104.

[0117] In S805, the information management unit 114 updates the cumulative evaluation value for each subject being managed. <Other Embodiments>

[0118] Although examples of embodiments have been described in detail above, the present invention can take the form of, for example, an information processing system, an information processing device, an information processing method, a program, or a recording medium (storage medium). Specifically, it may be applied to a system consisting of multiple devices (for example, a host computer, an interface device, an imaging device, a web application, etc.), or to a device consisting of a single device.

[0119] Furthermore, it goes without saying that the object of the present invention is achieved as follows: a recording medium (or storage medium) containing program code (computer program) of software that realizes the functions of the embodiments described above is supplied to a system or device. Needless to say, such storage medium is a computer-readable storage medium. The computer (or CPU or MPU) of the system or device then reads and executes the program code stored on the recording medium. In this case, the program code read from the recording medium itself realizes the functions of the embodiments described above, and the recording medium containing that program code constitutes the present invention. <Other>

[0120] The above-described embodiment includes the following configuration. (Composition 1) A recognition method for recognizing a subject from captured image data, An evaluation value determination means that determines an evaluation value based on the recognition result of the subject recognized by the recognition means, Priority determination means for determining the priority of each subject according to the evaluation value determined by the evaluation value determination means, The system includes a means for acquiring usage information for each subject recognized by the recognition means, The evaluation value determination means is an information processing device characterized by updating the evaluation value for each subject in accordance with the usage information. (Configuration 2) The information processing device according to configuration 1, wherein the recognition means is characterized by recognizing the face of the subject and the shooting scene. (Composition 3) The information processing device according to configuration 2, characterized in that the aforementioned shooting scene is at least one piece of information: the subjects appearing together, the location, the actions of the subjects, and the time of day the shooting took place. (Composition 4) The information processing apparatus according to configuration 2, wherein the recognition means recognizes at least one of the image quality, facial expression, face size, and face orientation of the subject. (Composition 5) The information processing device according to any one of configurations 1 to 4, characterized in that the captured image data is data obtained from a shooting device that photographs a subject. (Composition 6) An information processing device according to any one of configurations 1 to 4, comprising a shooting control means for shooting subjects whose priority has been determined by the priority determination means, wherein the shooting control means creates a still image or video by extracting a portion of the shooting image data based on the evaluation value for each subject recognized by the recognition means. (Composition 7) The information processing apparatus according to configuration 6, further comprising an assignment unit that assigns caption information to the aforementioned captured image data based on the subject and shooting scene recognized by the recognition means. (Composition 8) The information processing device according to configuration 6, comprising an information management means for managing together information on an accumulated evaluation value obtained by accumulating the evaluation value for each subject with respect to the captured image data, and the usage information, wherein the shooting control means photographs each subject until the accumulated evaluation value corresponding to the usage information for each subject reaches a predetermined value. (Composition 9) The information processing device according to configuration 8, wherein the information management means registers subjects to be photographed and adjusts the cumulative evaluation value according to the increase or decrease in the number of subjects to be photographed. (Composition 10) The information processing device according to any one of configurations 1 to 4, wherein the means for acquiring the usage information acquires the usage information for each user based on the usage history of the Web service. (Composition 11) The information processing apparatus according to any one of configurations 1 to 4, characterized in that the captured image data is data obtained from recorded video data of a subject that has already been captured. (Composition 12) A recognition method for recognizing a subject from captured image data, An evaluation value determination means that determines an evaluation value based on the recognition result of the subject recognized by the recognition means, Priority determination means for determining the priority of each subject according to the evaluation value determined by the evaluation value determination means, The system includes a means for acquiring usage information for each subject recognized by the recognition means, The evaluation value determination means is an information processing system characterized by updating the evaluation value for each subject in accordance with the usage information. (Composition 13) A recognition step that recognizes the subject from the captured image data, An evaluation value determination step in which an evaluation value is determined based on the recognition result of the subject recognized in the above recognition step, A priority determination step in which the priority of each subject is determined according to the evaluation value determined in the evaluation value determination step, The system includes a usage information acquisition step which acquires usage information for each subject recognized in the recognition step, The evaluation value determination step is characterized by updating the evaluation value for each subject according to the usage information.

Claims

1. A recognition method for recognizing a subject from captured image data, An evaluation value determination means that determines an evaluation value based on the recognition result of the subject recognized by the recognition means, Priority determination means for determining the priority of each subject according to the evaluation value determined by the evaluation value determination means, The system includes a means for acquiring usage information for each subject recognized by the recognition means, The evaluation value determination means is an information processing device characterized by updating the evaluation value for each subject in accordance with the usage information.

2. The information processing apparatus according to claim 1, wherein the recognition means recognizes the face of the subject and the shooting scene.

3. The information processing device according to claim 2, characterized in that the aforementioned shooting scene is at least one piece of information: the subjects appearing together, the location, the actions of the subjects, and the time of day the shooting took place.

4. The information processing apparatus according to claim 2, characterized in that the recognition means recognizes at least one of the image quality of the subject, facial expression, face size, and face orientation.

5. The information processing apparatus according to any one of claims 1 to 4, characterized in that the captured image data is data obtained from a shooting device that photographs a subject.

6. The information processing apparatus according to any one of claims 1 to 4, comprising a shooting control means for shooting subjects whose priority has been determined by the priority determination means, wherein the shooting control means creates a still image or video by extracting a portion of the shooting image data based on the evaluation value for each subject recognized by the recognition means.

7. The information processing apparatus according to claim 6, further comprising a captioning unit that adds caption information to the captured image data based on the subject and shooting scene recognized by the recognition means.

8. The information processing apparatus according to claim 6, comprising an information management means for managing together information of an accumulated evaluation value obtained by accumulating the evaluation value for each subject with respect to the captured image data, and the usage information, wherein the shooting control means photographs each subject until the accumulated evaluation value corresponding to the usage information for each subject reaches a predetermined value.

9. The information processing apparatus according to claim 8, characterized in that the information management means registers subjects to be photographed and adjusts the cumulative evaluation value according to the increase or decrease in the number of subjects to be photographed.

10. The information processing apparatus according to any one of claims 1 to 4, characterized in that the means for acquiring the usage information acquires the usage information for each user based on the usage history of the Web service.

11. The information processing apparatus according to any one of claims 1 to 4, characterized in that the captured image data is data obtained from recorded video data of a subject that has already been captured.

12. A recognition method for recognizing a subject from captured image data, An evaluation value determination means that determines an evaluation value based on the recognition result of the subject recognized by the recognition means, Priority determination means for determining the priority of each subject according to the evaluation value determined by the evaluation value determination means, The system includes a means for acquiring usage information for each subject recognized by the recognition means, The evaluation value determination means is an information processing system characterized by updating the evaluation value for each subject in accordance with the usage information.

13. A recognition step that recognizes the subject from the captured image data, An evaluation value determination step in which an evaluation value is determined based on the recognition result of the subject recognized in the above recognition step, A priority determination step in which the priority of each subject is determined according to the evaluation value determined in the evaluation value determination step, The system includes a usage information acquisition step which acquires usage information for each subject recognized in the recognition step, The evaluation value determination step is characterized by updating the evaluation value for each subject according to the usage information.

Images