Image processing apparatus

Abstract

A database in which an image of a user and state information are stored in association with each other is generated.SOLUTION: The image processing apparatus includes an acquisition unit that acquires captured image data in which a captured image of a user and a capturing time of the captured image are associated with each other and vital data in which biological information of the user and a detection time of the biological information are associated with each other, and a processing unit that generates user state data in which state information indicating a state of the user is calculated for each detection time based on the vital data and the detection time, and stores the captured image data and the user state data in a storage unit in a time-series manner by associating the capturing time and the detection time with each other.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image processing device. [Background technology]

[0002] An apparatus has been disclosed that generates an album of a user using a plurality of images of the user (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2004-258920 Summary of the Invention [Problem to be solved by the invention]

[0004] As an album generating device such as that described in Patent Document 1, there is a demand for a device that can generate a database that stores, in association with each other, images of a user and information indicating the user's state, such as a stress level.

[0005] The present invention has been made in view of the above, and has as its object to provide an image processing device capable of generating a database in which images of a user and status information are stored in association with each other. [Means for solving the problem]

[0006] The image processing device of the present invention includes an acquisition unit that acquires captured image data in which a captured image of a user is associated with the capture time of the captured image, and vital data in which the user's biometric information is associated with the detection time of the biometric information, and a processing unit that generates user status data by calculating status information indicating the user's status for each detection time based on the vital data and the detection time, and associates the captured image data with the user status data in chronological order by matching the capture time with the detection time, and stores the image data in a memory unit. [Effects of the Invention]

[0007] According to the present invention, it is possible to generate a database that stores images of users in association with their status information. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a functional block diagram showing an example of an image processing apparatus according to this embodiment. [Figure 2] FIG. 2 is a diagram showing an example of an operation for storing captured image data and user state data in a storage unit. [Figure 3] FIG. 3 is a diagram schematically illustrating an example of the correspondence between captured image data and user state data. [Figure 4] FIG. 4 is a diagram schematically illustrating an example of the correspondence between captured image data and user state data. [Figure 5] FIG. 5 is a diagram illustrating the specific period set by the period setting unit. [Figure 6] FIG. 6 is a flowchart showing an example of processing using the image processing device according to this embodiment. [Figure 7] FIG. 7 is a flowchart showing another example of processing using the image processing device according to this embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, an embodiment of an image processing device according to the present invention will be described with reference to the drawings. However, the present invention is not limited to this embodiment. Furthermore, the components in the following embodiments include those that are easily replaceable by those skilled in the art, or those that are substantially the same.

[0010] Fig. 1 is a functional block diagram showing an example of an image processing device 100 according to this embodiment. The image processing device 100 shown in Fig. 1 generates a composite image of a user based on one or more captured images of the user. As shown in Fig. 1, the image processing device 100 includes a communication unit 10, a control unit 20, and a storage unit 30.

[0011] The communication unit 10 performs wired or wireless communication with an external device and includes an interface such as a network interface card.

[0012] The control unit 20 performs various information processing. The control unit 20 includes a processor such as a CPU (Central Processing Unit) and memories such as a ROM (Read Only Memory) and a RAM (Random Access Memory). The control unit 20 includes an acquisition unit 21 and a processing unit 22.

[0013] The acquisition unit 21 acquires captured image data and vital data. The captured image data is data in which a captured image of the user is associated with the capture time of the captured image. The vital data is data in which biometric information of the user is associated with the detection time of the biometric information. It is preferable that the biometric information is not immutable information such as a fingerprint, but information whose value changes in response to external stimuli. Examples of such biometric information include heart rate, blood pressure, pulse wave, skin temperature, sweat rate, brain waves, and blood flow.

[0014] The processing unit 22 includes a storage processing unit 23, an extraction unit 24, a period setting unit 25, a background clothing processing unit 26, a facial expression processing unit 27, and a synthesis unit 28.

[0015] The storage processing unit 23 stores the captured image data and vital data acquired by the acquisition unit 21 in the storage unit 30. The storage processing unit 23 generates user condition data by calculating user condition information for each detection time based on the vital data and the detection time. In this embodiment, a stress level will be used as an example of the condition information. In this embodiment, the stress level is a value indicating the degree of reaction occurring in the user's body due to an external stimulus or the like. The storage processing unit 23 associates the captured image data and the user condition data in chronological order by associating the capture time with the detection time, and stores them in the storage unit 30.

[0016] The extraction unit 24 selects one of the photographed image data and the user status data to extract the other data corresponding to the selected one. In other words, the extraction unit 24 can extract user status data corresponding to the photographed image data or photographed image data corresponding to the user status data by utilizing the correspondence between the photographed image data and the user status data stored in association with each other in the storage unit 30.

[0017] The period setting unit 25 sets a specific period based on the user's settings, etc. The specific period is a period for chronologically identifying the captured image data and user status data to be used for generating a composite image from among the captured image data and user status data stored in the storage unit 30. In other words, the image processing device 100 generates a composite image of the user using the captured image data captured within the specific period set by the period setting unit 25 and the user status data detected within the specific period from among the captured image data and user status data stored in the storage unit 30.

[0018] The period setting unit 25 determines in chronological order whether an abnormality in the user's condition is detected in the captured images. Examples of abnormal conditions include, for example, abnormalities due to illness. Specific examples include a state in which a change in appearance, such as emaciation, occurs due to illness or fighting an illness, and a state in which abnormal values ​​in vital data continue for a long period of time. If the period setting unit 25 determines that an abnormal condition is not detected, it sets a specific period whose end point is the time of determination and whose start point is a time that is a predetermined period back from the end point. If the period setting unit 25 determines that an abnormal condition is detected, it sets a specific period whose end point is the time when the abnormal condition was detected and whose start point is a time that is a predetermined period back from the end point.

[0019] The background clothing processing unit 26 sets the user's clothing and background for the composite image based on the user's clothing and background included in the images taken during a specific period and the stress level corresponding to the images. The background clothing processing unit 26 extracts and classifies the user's clothing and background included in the images taken during a specific period using image processing. The background clothing processing unit 26 calculates the frequency for each clothing and background classification. The background clothing processing unit 26 calculates a first reference value by weighting the calculated clothing and background frequencies by the reciprocal of the stress level. The background clothing processing unit 26 sets the clothing and background with the highest calculated first reference value as the clothing and background for the composite image.

[0020] The facial expression processing unit 27 sets a facial expression of the user for a composite image based on the facial expressions of the user included in the images taken during a specific period and the stress level corresponding to the images. The facial expression processing unit 27 extracts and classifies the facial expressions of the user included in the images taken during a specific period. The facial expression processing unit 27 calculates the frequency for each classification of facial expression. The facial expression processing unit 27 calculates a second reference value by weighting the calculated frequency of the facial expression by the reciprocal of the stress level. The facial expression processing unit 27 sets the facial expression with the highest calculated second reference value as the facial expression for the composite image.

[0021] The synthesis unit 28 generates a synthetic image of the user using the user's clothing and background that have been set, and the user's facial expression.

[0022] The storage unit 30 stores various programs, information such as data, applications, etc. for performing processing in the control unit 20. The storage unit 30 includes storage such as an HDD (Hard Disk Drive) and an SSD (Solid State Drive). The storage unit 30 stores the above-mentioned captured image data and user state data in a state where they are associated in chronological order.

[0023] The memory unit 30 stores an image processing program that causes a computer to execute, for example, a process of acquiring captured image data in which a captured image of a user is associated with the capture time of the captured image, and vital data in which the user's biometric information is associated with the detection time of the biometric information, and a process of generating user status data in which the user's status information (stress level) is calculated for each detection time based on the vital data and the detection time, and by associating the capture time with the detection time, the captured image data and the user status data are associated in chronological order and stored in the memory unit.

[0024] The memory unit 30 also stores an image processing program that causes the computer to further execute the following processes in the image processing program: setting the user's clothing and background based on the user's clothing and background included in the captured image taken during a specific period and status information (stress level) corresponding to the captured image; setting the user's facial expression based on the user's facial expression included in the captured image taken during a specific period and status information (stress level) corresponding to the captured image; and generating a composite image of the user using the set user's clothing and background and the user's facial expression.

[0025] In the image processing device 100, the processor in the processing unit 22 reads out various programs and loads them into memory, thereby executing information processing corresponding to the functions of the above-mentioned units. Examples of the various programs include programs received by the communication unit 10, programs stored in the storage unit 30, and programs recorded on an external recording medium. The image processing device 100 functions as an information processing device (computer) that executes various information processes. Note that the various programs may be executed by an information processing device other than the image processing device 100, or the image processing device 100 and the other information processing device may cooperate to execute the various programs.

[0026] Next, an example of the operation of the image processing device 100 configured as above will be described.

[0027] First, an operation for storing captured image data and user condition data in the storage unit 30 will be described. FIG. 2 is a diagram showing an example of an operation for storing captured image data and user condition data in the storage unit 30. As shown in FIG. 2, first, captured image data D1 and vital data D2 are input to the image processing device 100. For example, the captured image data D1 can be data including a captured image captured by a capturing device 40 such as a camera and a capturing time indicating the time when the captured image was captured. The vital data D2 can be data including biometric information detected by a biometric information detection device 50 and the detection time when the biometric information was detected.

[0028] The acquisition unit 21 acquires the input photographed image data D1 and vital data D2. The storage processing unit 23 first generates user condition data D3 based on the biometric information and detection time of the acquired vital data D2. The storage processing unit 23, for example, compares the detected value of the biometric information with the normal value of the biometric information, and can calculate the stress level as a multi-level score based on the comparison result. The storage processing unit 23 generates the user condition data D3 by associating the calculated stress level with the detection time of the vital data D2 used to calculate the stress level.

[0029] After generating the user state data D3, the storage processing unit 23 associates the captured image data D1 and the user state data D3 in chronological order by associating the capture time with the detection time, and stores the data in the storage unit 30.

[0030] 3 and 4 are diagrams schematically showing an example of the correspondence between captured image data and user status data. For example, as shown in Fig. 3, the storage processing unit 23 can store captured image data D1 and user status data D3 in one-to-one correspondence in the storage unit 30. In this case, the storage processing unit 23 extracts the detection time of the user status data D3 that matches or has the closest value to the shooting time of one piece of captured image data D1, and associates the user status data D3 of the extracted detection time with the captured image data D1.

[0031] Furthermore, for example, as shown in Fig. 4, the storage processing unit 23 can store the captured image data and the user state data in the storage unit 30 so that they are arranged along a single time axis. In the example shown in Fig. 4, captured image data D11 at a capture time t1, user state data D32 at a detection time t2, user state data D33 at a detection time t3, captured image data D14 at a capture time t4, user state data D35 at a detection time t5, captured image data D16 at a capture time t6, ... are stored in the storage unit 30 so as to be arranged along a single time axis.

[0032] The manner in which the photographed image data D1 and the user status data D3 are associated with each other is not limited to the above, and the photographed image data D1 and the user status data D3 may be associated with each other in a manner different from the above.

[0033] From this state, when one of the captured image data D1 and the user status data D3 is designated by a user operation or the like, the extraction unit 24 can select the one of the data and extract the other data corresponding to the selected one. By utilizing the correspondence between the captured image data D1 and the user status data D3 stored in association with each other in the storage unit 30, the extraction unit 24 can extract the user status data D3 corresponding to the captured image data D1 when the user designates the captured image data D1. Furthermore, when the user designates the user status data D3, the extraction unit 24 can extract the captured image data D1 corresponding to the user status data D3.

[0034] Next, an operation for generating a composite image of a user using the captured image data D1 and the user state data D3 stored in the storage unit 30 will be described.

[0035] The period setting unit 25 sets a specific period based on the user's settings, etc. The specific period is a period for chronologically identifying the captured image data D1 and the user state data D3 to be used for generating a composite image from among the captured image data D1 and the user state data D3 stored in the storage unit 30. That is, the image processing device 100 generates a composite image of the user using the captured image data D1 captured within the specific period set by the period setting unit 25 and the user state data D3 detected within the specific period from among the captured image data D1 and the user state data D3 stored in the storage unit 30.

[0036] FIG. 5 is a diagram illustrating a specific period set by the period setting unit 25. The period setting unit 25 determines, in chronological order, whether or not an abnormality in the user's condition is detected in the captured images of each piece of captured image data D1. If the period setting unit 25 determines that an abnormality in the user's condition is not detected (ST1), it sets a specific period TA whose end point is time t51, which is the time of determination, and whose start point is time t52, which is a predetermined period T prior to the end point. On the other hand, if the period setting unit 25 determines that an abnormality in the user's condition is detected in the captured images of each piece of captured image data D1 (ST2), it sets a specific period TB whose end point is time t53, which is the time at which detection of the abnormal condition began in the chronological order, and whose start point is time t54, which is a predetermined period T prior to the end point. In this case, the period during which the abnormality in the user's condition was detected, i.e., the period TC from time t53, when detection of the abnormal condition began in the chronological order, to time t51, when the determination was made, is not included in the specific period.

[0037] The background clothing processor 26 sets the user's clothing and background based on the user's clothing and background included in images captured during a specific period and the stress level corresponding to the captured image. The background clothing processor 26 extracts and classifies the user's clothing and background included in images captured during a specific period. For example, the background clothing processor 26 can classify clothing by category (jacket, pants, skirt, scarf, etc.) and color. The background clothing processor 26 may identify and classify each piece of clothing individually. Furthermore, the background clothing processor 26 can classify backgrounds by category (location, brightness, etc.). The categories classified by the background clothing processor 26 are not limited to the above and may be classified based on other categories. The background clothing processor 26 calculates the frequency for each clothing and background category. The background clothing processor 26 calculates a first reference value by weighting the calculated frequency of clothing and background by the inverse of the stress level. The background clothing processor 26 sets the clothing and background based on the calculated first reference value. For example, background clothing processing unit 26 can set the clothing and background for which the calculated first reference value is highest or lowest as the clothing and background for the composite image. Background clothing processing unit 26 can also set the clothing and background for which the calculated first reference value is below a predetermined threshold, or the clothing and background for which the calculated first reference value is equal to or greater than a predetermined threshold, as the clothing and background for the composite image. Background clothing processing unit 26 stores the classification results for the clothing and background, the setting results for the composite image, and the like in storage unit 30.

[0038] The facial expression processing unit 27 sets a user's facial expression for a composite image based on the user's facial expression included in images captured during a specific period and the stress level corresponding to the captured image. The facial expression processing unit 27 extracts and classifies the user's facial expression included in images captured during a specific period. The facial expression processing unit 27 can classify facial expressions into types such as smiling, crying, and expressionless faces. The facial expression processing unit 27 may also classify facial expressions based on features of parts of the face, such as the shape of the eyes, the shape of the mouth, and wrinkles on the forehead and cheeks. The facial expression processing unit 27 calculates the frequency for each classification of facial expression. The facial expression processing unit 27 calculates a second reference value by weighting the calculated frequency of the facial expression by the reciprocal of the stress level. The facial expression processing unit 27 sets the facial expression based on the calculated second reference value. For example, the facial expression processing unit 27 can set the facial expression for which the calculated second reference value is highest or lowest as the facial expression for the composite image. Furthermore, the facial expression processing unit 27 can set, as the facial expression for the composite image, a facial expression for which the calculated second reference value is less than a predetermined threshold value or a facial expression for which the calculated second reference value is equal to or greater than a predetermined threshold value. The facial expression processing unit 27 stores the classification results for the facial expressions, the setting results for the composite image, etc. in the storage unit 30.

[0039] The synthesis unit 28 generates a synthetic image of the user using the user's clothing and background that have been set, and the user's facial expression.

[0040] Fig. 6 is a flowchart showing an example of processing using the image processing device 100 according to this embodiment. As shown in Fig. 6, the acquisition unit 21 acquires photographed image data D1 and vital data D2 (step S101).

[0041] The storage processing unit 23 generates user condition data D3 based on the acquired vital data D2 and the detection time (step S102).

[0042] The storage processing unit 23 stores the acquired photographed image data D1 and the generated user state data D3 in association with each other in chronological order in the storage unit 30 (step S103).

[0043] The extraction unit 24 determines whether one of the captured image data D1 and the user status data D3 has been selected (step S104). If it is determined that one of the captured image data D1 and the user status data D3 has not been selected (No in step S104), the process of step S104 is repeated. If it is determined that one of the captured image data D1 and the user status data D3 has been selected (Yes in step S104), the extraction unit 24 extracts the other data corresponding to the one of the data (step S105).

[0044] Fig. 7 is a flowchart showing another example of processing using the image processing device 100 according to this embodiment. The processing shown in Fig. 7 is processing for generating a composite image of a user based on captured image data D1 and user state data D3.

[0045] As shown in FIG. 7, when a period for setting a specific period is input by the user (step S201), the period setting unit 25 determines in chronological order whether or not an abnormality in the user's condition contained in the captured image is detected during the period set by the user (step S202).

[0046] If the period setting unit 25 determines that an abnormality in the state is detected (Yes in step S202), it sets a specific period whose end point is the time when the detection of the abnormality in the state started and whose start point is a time that is a predetermined period back from the end point (step S203).If the period setting unit 25 determines that an abnormality in the state is not detected (No in step S202), it sets a specific period whose end point is the time of the determination and whose start point is a time that is a predetermined period back from the end point (step S204).

[0047] After the specific period is set, the background clothing processing unit 26 sets the user's clothing and background based on the user's clothing and background included in the captured images taken during the specific period and the stress level corresponding to the captured images (step S205).

[0048] The facial expression processor 27 sets the facial expression of the user for the composite image based on the facial expression of the user included in the photographed image taken during a specific period and the stress level corresponding to the photographed image (step S206).

[0049] The synthesis unit 28 generates a synthetic image of the user using the set clothing and background of the user and the facial expression of the user (step S207).

[0050] As described above, the image processing device 100 according to this embodiment includes an acquisition unit 21 that acquires captured image data D1 in which a captured image of a user is associated with the time the captured image was taken, and vital data D2 in which the user's biometric information is associated with the time the biometric information was detected, and a processing unit 22 that generates user status data D3 that calculates the user's stress level for each detection time based on the vital data D2 and the detection time, and associates the captured image data D1 with the user status data D3 in chronological order by associating the capture time with the detection time and stores them in a memory unit.

[0051] According to this configuration, by correlating the captured image data D1 with the user condition data D3 generated based on the vital data D2 and storing them in the memory unit 30, it is possible to easily generate a database in which the captured image is associated with the stress level.

[0052] In the image processing device 100 according to this embodiment, the processing unit 22 selects one of the captured image data D1 and the user state data D3, thereby extracting the other data corresponding to the one data.

[0053] According to this configuration, when one piece of data is selected, the other piece of data can be extracted using the correspondence between the captured image data D1 and the user state data D3.

[0054] In the image processing device 100 according to this embodiment, the processing unit 22 sets the user's clothing and background based on the user's clothing and background included in the captured images taken during a specific period and the stress level corresponding to the captured images, sets the user's facial expression based on the user's facial expression included in the captured images taken during a specific period and the stress level corresponding to the captured images, and generates a composite image of the user using the set user's clothing and background and the user's facial expression.

[0055] According to this configuration, the user's clothing, background, and facial expression can be set using the correspondence between the captured image and the stress level, and a composite image of the user can be generated using the setting results, thereby generating a composite image of the user according to the stress level.

[0056] In the image processing device 100 according to this embodiment, the processing unit 22 extracts and classifies the user's clothing and background, and the user's facial expression, contained in the captured images taken during a specific period, calculates the frequency for each clothing and background category and each facial expression category, and generates a composite image using the clothing and background for which the first reference value, obtained by weighting the calculated clothing and background frequencies by the inverse of the stress level, is the highest, and the facial expression for which the second reference value, obtained by weighting the calculated facial expression frequencies by the inverse of the stress level, is the highest.

[0057] According to this configuration, a composite image of the user can be generated according to the stress level and frequency.

[0058] In the image processing device 100 according to this embodiment, the processing unit 22 determines in chronological order whether or not an abnormality in the user's condition is detected in the captured image, and if it determines that no abnormality in the condition is detected, it sets a specific period with the time of determination as the end point and a time a predetermined period back from the end point as the start point, and if it determines that an abnormality in the condition is detected, it sets a specific period with the time at which the abnormality in the condition was detected as the end point and a time a predetermined period back from the end point as the start point.

[0059] According to this configuration, the specific period can be set so as not to include images captured after the user's condition has changed.

[0060] The technical scope of the present invention is not limited to the above-described embodiment, and appropriate modifications can be made without departing from the spirit of the present invention. For example, in the above-described embodiment, the stress level is used as an example of the user's state information, but the state information is not limited to the stress level and may be a value indicating another state that can be calculated based on the vital data D2, such as a value indicating the user's enjoyment or a value indicating the user's sense of elation. [Explanation of symbols]

[0061] D1, D11, D14, D16...captured image data, D2...vital data, D3, D32, D33, D35...user status data, T, TC...period, TA, TB...specific period, t51, t52, t53, t54...time, 10...communication unit, 20...control unit, 21...acquisition unit, 24...extraction unit, 22...processing unit, 23...storage processing unit, 25...period setting unit, 26...background clothing processing unit, 27...facial expression processing unit, 28...combination unit, 30...storage unit, 40...capturing device, 50...biometric information detection device, 100...image processing device

Claims

1. an acquisition unit that acquires captured image data in which a captured image of a user is associated with a capture time of the captured image, and vital data in which biometric information of the user is associated with a detection time of the biometric information; a processing unit that generates user status data by calculating status information indicating the status of the user for each detection time based on the vital data and the detection time, and that associates the photographed image data with the user status data in chronological order by associating the photographed image data with the detection time, and stores the associated data in a storage unit; An image processing device comprising:

2. The processing unit selects one of the photographed image data and the user state data, thereby extracting the other data corresponding to the one of the photographed image data and the user state data. The image processing device according to claim 1 .

3. The processing unit setting the user's clothing and background based on the user's clothing and background included in the photographed image photographed during a specific period and the status information corresponding to the photographed image; setting a facial expression of the user based on a facial expression of the user included in the photographed image photographed during the specific period and the status information corresponding to the photographed image; A composite image of the user is generated using the user's clothing and background and the user's facial expression. The image processing device according to claim 1 .

4. The processing unit extracts and classifies the clothing and background of the user and the facial expression of the user included in the captured images captured during the specific period, calculates a frequency for each classification of the clothing and background and each classification of the facial expression, and generates the composite image using the clothing and background for which a first reference value, which is obtained by weighting the calculated frequencies of the clothing and background by the reciprocal of the state information, is the highest, and the facial expression for which a second reference value, which is obtained by weighting the calculated frequencies of the facial expression by the reciprocal of the state information, is the highest. The image processing device according to claim 3 .

5. The processing unit determining whether an abnormality in the user's condition is detected in the captured image in a time series manner; If it is determined that no abnormality in the state is detected, the specific period is set to have the time of determination as the end point and the time preceding the end point by a predetermined period as the start point, When it is determined that the abnormality in the state is detected, the specific period is set so that the end point is the time when the detection of the abnormality in the state starts, and the start point is a time that is a predetermined period before the end point. The image processing device according to claim 3 .

Images