Image processing system

Abstract

To provide an image processing system capable of generating an image in which a level of privacy of a user is reflected depending on a call partner.SOLUTION: An image processing system comprises one or more processors. The one or more processors are configured so as to execute processing of acquiring a first image on a user, mode selection processing of selecting one mode from a plurality of modes on the basis of information on a communication partner of the user, and image generation processing of generating a second image from the first image depending on the selected mode. The plurality of modes include a first mode for making the first image be the second image, a second mode for making processed image in which an information amount on the user is smaller than that of the first image be the second image, and a third mode for making an illustration image generated on the basis of information on the user be the second image.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] The present disclosure relates to a technology for acquiring an image from a user and appropriately processing it.

Background Art

[0002] Patent Document 1 discloses an electronic device including a processing unit that divides a video showing a user into a plurality of regions and generates a video in which regions not selected by the user among the plurality of regions are distorted according to the call time and the amount of conversation with the call partner, and a communication unit that transmits the generated video to the electronic device of the call partner.

[0003] In addition, as documents showing the technical level of the technical field related to the present disclosure, there are the following Patent Document 2 and Patent Document 3.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Patent Document 3

Summary of the Invention

Problems to be Solved by the Invention

[0005] According to the technology disclosed in Patent Document 1, the image displayed to the call partner is an image in which a region not selected by the user (for example, a region where the face is reflected) is distorted according to the call time and the amount of conversation with the call partner.

[0006] However, there is a risk that the privacy of the user cannot be sufficiently protected by simply gradually changing the image by dynamic elements such as the call time and the amount of conversation.

[0007] One object of the present disclosure is to provide a technology capable of preliminarily generating an image in which the level of a user's privacy is reflected according to a communication partner. **Means for Solving the Problems**

[0008] One aspect of the present disclosure relates to an image processing system.

[0009] The image processing system according to the present disclosure includes one or more processors, and the one or more processors perform a process of acquiring a first image related to a user, a mode selection process of selecting one mode from a plurality of modes based on information related to a communication partner of the user, and an image generation process of generating a second image from the first image according to the selected mode. The plurality of modes include a first mode in which the first image is the second image, a second mode in which a processed image with a reduced amount of information related to the user from the first image is the second image, and a third mode in which an illustrated image generated based on information related to the user is the second image. **Brief Description of the Drawings**

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

[0011] Hereinafter, with reference to the drawings, a preferred example of an embodiment will be described.

[0012] 1. Configuration The image processing system according to this embodiment is a system that acquires an image from a user and appropriately processes it. Various modes of the image processing system are assumed. For example, as one of the image processing systems, an image communication system that provides an image-based communication function between users can be cited. The image processing system as an image communication system acquires an image of a user captured by a camera and executes a process of presenting the acquired image to a communication partner. Examples of other image processing systems include a system that provides an SNS (Social Networking Service) using images, a system that provides an application for sharing images between users, and the like. Hereinafter, an image communication system, which is one aspect of the image processing system according to this embodiment, will be described.

[0013] FIG. 1 is a conceptual diagram showing an outline of the configuration of an image communication system 10 according to this embodiment.

[0014] The image communication system 10 includes a server 100. The server 100 communicates with user terminals 200 of users who utilize the image communication function and executes processes related to the image communication function. In FIG. 1, users 1a and 1b who utilize the image communication function with each other are shown. In this case, the server 100 communicates with the user terminal 200 of user 1a and the user terminal 200 of user 1b. Note that the communication between the server 100 and the user terminal 200 is performed via, for example, the Internet.

[0015] In the case shown in FIG. 1, the server 100 receives, as an image related to user 1a, an image of user 1a captured by the camera 210 (captured image) from the user terminal 200 of user 1a. Then, based on the received captured image, the server 100 generates an image (display image) to be presented to user 1b, who is the communication partner of user 1a, from the captured image, and transmits the generated display image to the user terminal 200 of user 1b. Then, the user terminal 200 of user 1b displays the display image received from the server 100 to user 1b. Similarly, the server 100 generates a display image to be presented to user 1a from the captured image of user 1b, and transmits the generated display image to the user terminal 200 of user 1a. In this way, in the image communication system 10, it is assumed that user 1a and user 1b communicate with each other via images.

[0016] FIG. 1 shows an example of the configuration of the user terminal 200. The user terminal 200 shown in FIG. 1 includes a camera 210, a display device 220, an input device 230, and an information processing device 240. The information processing device 240 is connected so as to be able to transmit and receive information with the camera 210, the display device 220, and the input device 230. For example, the information processing device 240 is electrically connected to these devices via cables.

[0017] The camera 210 captures the user and outputs a captured image. The input device 230 is an HMI (Human Machine Interface) that receives the user's operations. Examples of the input device 230 include a keyboard, a switch, a touch panel, an operation panel, and the like. The information processing device 240 includes a processor and executes various processes. At least, the information processing device 240 executes a process of controlling the display of the display device 220. Further, the information processing device 240 may be provided with a communication device that communicates with the server 100 to transmit and receive information. For example, the information processing device 240 may be provided with a communication device that connects to the Internet. The display device 220 performs display according to the processes executed by the information processing device 240. The form of the display device 220 is not particularly limited. By the information processing device 240 executing the processes, transmission of the captured image to the server 100, reception of the display image from the server 100, and display of the display image are realized in the user terminal 200.

[0018] Note that the user terminal 200 may typically be a general personal computer installed with a predetermined application. In this case, the personal computer may be configured to operate as the user terminal 200 of the image communication system 10 on a predetermined application.

[0019] Hereinafter, with reference to FIG. 2, the configuration of the server 100 will be described. FIG. 2 is a block diagram showing an example of the configuration of the server 100.

[0020] Server 100 includes one or more processors 110 (hereinafter simply referred to as "processor 110"), one or more storage devices 120 (hereinafter simply referred to as "storage device 120"), and a communication interface 130. The processor 110 executes various processes. The processor 110 can be composed of a CPU (Central Processing Unit) including, for example, an arithmetic unit and registers. The storage device 120 is coupled to the processor 110 and stores various information necessary for the execution of the processes by the processor 110. The storage device 120 can be composed of a recording medium such as a ROM (Read Only Memory), a RAM (Random Access Memory), an HDD (Hard Disk Drive), or an SSD (Solid State Drive). The communication interface 130 communicates with devices external to the server 100 to transmit and receive information. The communication interface 130 is, for example, a device that connects to the Internet to transmit and receive information. At least the communication interface 130 communicates with the user terminal 200.

[0021] The storage device 120 stores a computer program 121, a captured image 122, one or more template models 123 (hereinafter simply referred to as "template model 123"), and user information 124.

[0022] The computer program 121 is composed of a plurality of instructions INS. By operating the processor 110 in accordance with the plurality of instructions INS, the execution of various processes is realized. Note that the computer program 121 may be stored in a computer-readable recording medium.

[0023] The captured image 122 is received from the user terminal 200. When a plurality of users are using the image communication function, the storage device 120 may be configured to manage the captured image 122 for each user.

[0024] The template model 123 is a model representing a human figure. As an example of the template model 123, a plain polygon model without texture information can be cited. However, it is desirable that the template model 123 has rigging performed on it. The template model 123 may include a plurality of models with different patterns. For example, the template model 123 may include a plurality of models with different expressions such as height, body type, facial features (arrangement of eyes, nose, mouth, ears, etc.), clothing, etc., and different numbers of polygons.

[0025] The user information 124 manages the information of each user who uses the image communication function. The user information 124 includes communication history information D10, communication content information D20, and intimacy information D30.

[0026] The communication history information D10 manages the history of communication date and time, communication duration, communication frequency, etc. with other users for each user.

[0027] The communication content information D20 manages the information about the content of the communication with other users for each user. The information about the content of the communication is, for example, information about keywords that appeared in the communication and the frequency of appearance of the keywords. Alternatively, it may be a feature amount calculated by a machine learning model using the data recording the content of the communication as input.

[0028] The processor 110 may be configured to execute a process of updating the communication history information D10 and the communication content information D20 upon receiving that the user has ended the communication.

[0029] The intimacy information D30 manages the intimacy between users. The intimacy is managed by a numerical value, for example. In this case, typically, the magnitude of the intimacy is the magnitude of the numerical value. The intimacy information D30 may be configured to manage the intimacy with respect to other users for each user, or may be configured to manage the intimacy for combinations of users.

[0030] The processor 110 may be configured to execute a process of receiving the setting of the intimacy level by the user. For example, the processor 110 receives the setting of the intimacy level from the user terminal 200 via the communication interface 130. Then, the processor 110 updates the intimacy information D30 according to the received setting of the intimacy level. In this case, the user terminal 200 may be configured such that the user can input the setting of the intimacy level by operating the input device 230. By configuring in this way, the user can freely set the intimacy level using the relationship with the communication partner as an index. Furthermore, it is also assumed that the user can set the intimacy level using the casualness or formality of the scene where the communication function is used as an index.

[0031] Alternatively, the processor 110 may be configured to execute a process of estimating the intimacy level based on the communication history information D10 or the communication content information D20. For example, the processor 110 estimates the intimacy level for a specific user from the length of the communication time and the number of communication times with that specific user. Also for example, the processor 110 estimates the intimacy level for a specific user from the number of appearances and the frequency of appearance of a predetermined keyword in the communication with that specific user. Also for example, the processor 110 estimates the intimacy level using a machine learning model with the calculated feature amount as an input. The processor 110 updates the intimacy information D30 according to the estimated intimacy level. By configuring in this way, the server 100 can manage an appropriate intimacy level according to the communication situation for each user. Also, it is possible to reduce the user's hesitation about giving the intimacy level to the communication partner.

[0032] As described above, the image communication system 10 according to the present embodiment is configured. As an example of other configurations, the template model 123 and user information 124 can also be configured to be managed by a storage device external to the server 100. For example, the template model 123 and user information 124 may be stored in a data server accessible by the server 100. In this case, the server 100 is configured to acquire the template model 123 and user information 124 from the data server via the communication interface 130.

[0033] Alternatively, as an example of other configurations, the image communication system 10 according to the present embodiment can also be configured to execute a process of generating a display image on the user terminal 200. In this case, the configuration of the server 100 shown in FIG. 2 may be applied to the configuration of the information processing device 240. Also in this case, the server 100 receives the display image from the user terminal 200 and transmits the received display image to the user terminal 200 of the communication partner. When adopting such a configuration, in the following description, the processor 110 shall be read as the processor of the information processing device 240.

[0034] 2. Process The processor 110 executes a process of generating a display image to be presented to the communication partner in the process related to the image communication function. Particularly in the process of generating the display image, the processor 110 selects one mode from a plurality of modes based on the information about the communication partner. Particularly, the processor 110 selects one mode from a plurality of modes based on the degree of intimacy with the communication partner. Then, the processor 110 generates a display image from the captured image 122 according to the selected mode.

[0035] The plurality of modes include a first mode in which the captured image 122 is used as the display image as it is, a second mode in which a processed image with the amount of information about the user reduced from the captured image 122 is used as the display image, and a third mode in which an illustration image generated based on the information about the user reflected in the captured image 122 is used as the display image. Further, the plurality of modes may include a fourth mode in which a specific image (for example, a still image selected by the user or a predefined default image) is used as the display image. These plurality of modes can be said to have a higher privacy level in the order of the fourth mode, the third mode, the second mode, and the first mode.

[0036] Hereinafter, the process of generating the display image executed by the processor 110 will be described.

[0037] FIG. 3 is a block diagram showing an example of the design of the execution process by the processor 110. The process of generating the display image in FIG. 3 is composed of a mode selection processing unit P100, a tracking processing unit P200, a user model generation processing unit P300, and a display image generation processing unit P400.

[0038] The mode selection processing unit P100 selects one mode from the plurality of modes based on the degree of intimacy with the communication partner. The selection of the mode may be performed by threshold determination regarding the degree of intimacy.

[0039] FIG. 4(A) shows an example of mode selection by threshold determination regarding the degree of intimacy. According to the example shown in FIG. 4(A), when the degree of intimacy is equal to or greater than the first threshold, the first mode is selected; when it is less than the first threshold and equal to or greater than the second threshold, the second mode is selected; when it is less than the second threshold and equal to or greater than the third threshold, the third mode is selected. Also, when the degree of intimacy is less than the third threshold, the fourth mode is selected. By selecting the mode in this way, it can be configured such that the lower the degree of intimacy, the higher the privacy level mode is selected.

[0040] The tracking processing unit P200 performs tracking of the user shown in the captured image 122 and generates tracking information. Typically, the tracking processing unit P200 performs face tracking and body tracking of the user shown in the captured image 122. For the processing related to tracking, suitable known techniques may be adopted.

[0041] When the mode selected by the mode selection processing unit P100 is the second mode, the user model generation processing unit P300 generates a user model that deforms and represents the user shown in the captured image 122. When the selected mode is the first mode, the third mode, or the fourth mode, the processing related to the user model generation processing unit P300 does not need to be executed.

[0042] An example of the user model 223 generated by the user model generation processing unit P300 is shown in FIG. 4(B). The user model generation processing unit P300 generates the user model 223 as follows, for example.

[0043] First, the user model generation processing unit P300 selects a template model 123. For example, the user model generation processing unit P300 selects a template model 123 with a pattern similar to the characteristics of the user shown in the captured image 122. Next, the user model generation processing unit P300 generates the texture of the user shown in the captured image 122. For example, the user model generation processing unit P300 recognizes the user's parts (eyes, nose, mouth, clothing, accessories, etc.) by image recognition of the captured image 122 and generates a texture for each part. Then, the user model generation processing unit P300 generates the user model 223 by incorporating the generated texture into the selected template model 123.

[0044] In (B) of FIG. 4, an example of the user model 223 with a low degree of deformation and an example of the user model 223 with a high degree of deformation are shown. The user model generation processing unit P300 may be configured to generate the user model 223 with a higher degree of deformation as the intimacy is smaller. For example, the user model generation processing unit P300 is configured to select the template model 123 and generate the texture using the magnitude of the intimacy as an index. In (B) of FIG. 4, two user models 223 with different degrees of deformation are shown, but the user model generation processing unit P300 may be configured to generate user models 223 with different degrees of deformation in more stages or continuously according to the magnitude of the intimacy.

[0045] Furthermore, the user model generation processing unit P300 may be configured to process the generated user model 223 according to the user's request. For example, the user model generation processing unit P300 may be configured to perform virtual makeup or add decorations to the generated user model 223 according to the user's request.

[0046] Referring to FIG. 3 again. The display image generation processing unit P400 generates a display image from the captured image 122 according to the selected mode. The display image generation processing unit P400 generates the display image as follows, for example.

[0047] When the selected mode is the first mode, the display image generation processing unit P400 uses the captured image 122 as the display image.

[0048] When the selected mode is the second mode, the display image generation processing unit P400 generates, as the display image, a processed image in which the user model 223 is operated according to the tracking information. Note that the background portion of the processed image may be equivalent to the captured image 122. Thereby, when the selected mode is the second mode, a processed image in which the user shown in the captured image 122 is deformed and expressed is generated as the display image.

[0049] Alternatively, when the selected mode is the second mode, the display image generation processing unit P400 may be configured to generate a display image by extracting the feature amount of the user reflected in the captured image 122 and combining the mesh data and the block image generated based on the extracted user feature amount. By applying such a configuration, it is also possible to generate a processed image in which the user reflected in the captured image 122 is deformed and expressed. In particular, in this case, the display image generation processing unit P400 can change the degree of deformation of the user in the processed image by adjusting the size of the mesh and the resolution of the block image.

[0050] When the selected mode is the third mode, the display image generation processing unit P400 generates an illustration image in which the user reflected in the captured image 122 is represented by an illustration as the display image. For example, the display image generation processing unit P400 generates an illustration image by combining similar part illustrations based on the features of the user reflected in the captured image 122. Also for example, the display image generation processing unit P400 generates an illustration image by using the captured image 122 as an input with a machine learning model.

[0051] When the selected mode is the fourth mode, the display image generation processing unit P400 uses a specific image as the display image.

[0052] FIG. 5 shows an example of the display image generated by the display image generation processing unit P400. As shown in FIG. 5, the display image generated by the display image generation processing unit P400 from the captured image 122 has a higher privacy level as the intimacy is smaller.

[0053] As described above, the processor 110 executes a process of generating a display image. Note that the processor 110 may be configured to store the generated user model 223 or illustration image in the storage device 120. Then, for communications performed during a certain period, the processor 110 may be configured to acquire the user model 223 or illustration image stored in the storage device 120 and generate a display image. Thereby, the number of times of executing the process of generating the user model 223 or illustration image can be reduced, and the processing load can be reduced.

[0054] FIG. 6 is a flowchart showing an example of a process executed by the processor 110. The flowchart shown in FIG. 6 starts, for example, upon receiving the start of communication.

[0055] In step S100, the processor 110 acquires the degree of intimacy of the user with the communication partner. The processor 110 may acquire the degree of intimacy by referring to the intimacy information D30, or may estimate the degree of intimacy based on the communication history information D10 or the communication content information D20.

[0056] Next, in step S200, the processor 110 selects one mode from a plurality of modes based on the magnitude of the acquired degree of intimacy.

[0057] Next, in step S500, the processor 110 generates a display image according to the selected mode. The processor 110 transmits the generated display image to the user terminal 200 of the communication partner via the communication interface 130. Thereafter, until the communication ends, the generation of the display image and the transmission of the generated display image are repeated (step S600). When the communication ends, the process ends.

[0058] In this way, the processor 110 executes a process of generating a display image. Also, by the processor 110 executing the process of generating a display image in this way, the method for generating a display image according to the present embodiment is realized.

[0059] 3. Effects As described above, according to the image communication system 10, one mode is selected from a plurality of modes based on information about the communication partner, and a display image is generated from the captured image 122 according to the selected mode. The plurality of modes include a first mode in which the captured image 122 is used as the display image as it is, a second mode in which a processed image with the amount of information about the user reduced from the captured image 122 (specifically, a processed image in which the user reflected in the captured image 122 is deformed and represented) is used as the display image, and a third mode in which an illustration image generated based on information about the user (specifically, an illustration image in which the user reflected in the captured image 122 is represented as an illustration) is used as the display image. Thus, according to the image communication system 10, display images with different privacy levels are generated according to information about the communication partner. Furthermore, according to the present embodiment, an image in which the privacy level of the user is reflected according to the communication partner can be generated in advance.

[0060] The information about the communication partner is, for example, the intimacy with the communication partner. According to the present embodiment, an image with a higher privacy level can be generated as the intimacy is lower. Thereby, the user's privacy can be appropriately protected according to the intimacy with the communication partner.

Explanation of Reference Numerals

[0061] 10 Image communication system, 100 Server, 110 Processor, 120 Storage device, 122 Captured image, 123 Template model, 200 User terminal, 220 Display device, D10 Communication history information, D20 Communication content information, D30 Intimacy information

Claims

1. An image processing system, comprising one or more processors, wherein the one or more processors are configured to: obtain a first image which is an image of a user captured by a camera; perform mode selection processing to select one mode from a plurality of modes based on information about the user's communication partner; perform image generation processing to generate a second image from the first image according to the selected mode; and are configured to execute the above operations, wherein the plurality of modes include: a first mode in which the first image is used as the second image; a second mode in which an image of a user model representing the user, which is generated by incorporating the texture of the user shown in the first image into a template model representing a humanoid, is used as the second image; a third mode in which an illustrated image of the user, which is generated by combining part illustrations similar to the user based on the characteristics of the user shown in the first image, is used as the second image; and the texture is generated for each part of the user recognized by image recognition, wherein the template model is a polygon model that does not incorporate texture information. An image processing system.

2. The image processing system according to claim 1, wherein the template model includes a plurality of models with different patterns, and the one or more processors are further configured to: in the second mode, select a model with a pattern similar to the characteristics of the user from the plurality of models, and use the selected model in the image generation processing. An image processing system configured as above.

3. The image processing system according to claim 1 or claim 2, wherein the information about the communication partner includes information about the intimacy of the user with respect to the communication partner, and in the mode selection processing, the one or more processors are configured to: select the first mode when the intimacy is equal to or greater than a first threshold; select the second mode when the intimacy is less than the first threshold and equal to or greater than a second threshold. An image processing system configured as above.

4. The image processing system according to claim 1 or claim 2, wherein the information about the communication partner includes information about the intimacy of the user with respect to the communication partner, ​ ​ ​ In the image generation process when the second mode is selected, the one or more processors are configured to increase the degree of deformation of the user in the image of the user model as the intimacy is smaller. Image processing system. **Claim 5** The image processing system according to claim 1, the one or more processors are further configured to execute a process of presenting the second image to the communication partner. Image processing system.

Images