Camera system and method for generating a person's gaze contact image view

Abstract

The present disclosure relates to a camera system (100) for generating a gaze contact image view of a person, wherein the person has a gaze direction deviating from a camera position, the camera system (100) comprising a camera (10a, 10b, 10c, 10d) configured to acquire image data of the person, a processing circuit (102a, 102b, 102c) configured to be operably connected to the camera (10a, 10b, 10c, 10d) and configured to acquire a camera view of the person from the camera position and to generate a synthetic camera view of the person seen from a virtual camera position. The present disclosure further relates to a method for generating a gaze contact image view of a person, wherein the person has a gaze direction deviating from a camera position, and to a processing circuit program product (500).

Description

Technical Field

[0001] This disclosure relates to a camera system for video telephony. Background Technology

[0002] During a normal conversation, the first person usually looks into the second person's eyes to get their attention and establish a connection.

[0003] During a video call, the first person looks into the eyes of the second person, but the camera isn't always positioned where the first person is looking, so it often appears as if the first person is looking away from the second person's eyes. For a better user experience, it's desirable to place the camera where the person is looking. However, this is often very difficult for various reasons.

[0004] Inside a vehicle, occupants should be looking at the road ahead, making it particularly difficult to place a camera where they are looking. Occupants may be looking in the direction through the windshield, making it difficult to position the camera there. Furthermore, the direction of gaze through the windshield changes over time (e.g., depending on traffic conditions around the vehicle), making it impossible to place the camera in all directions of gaze. Occupants may also be looking around the vehicle through rear or side windows, or at other passengers in mirrors or on the rearview camera display.

[0005] Another example is when multiple people in a meeting room are using a single camera setup for video conferencing with someone remotely. It might be more natural for people in the meeting room to not always be looking at the camera setup, but rather at other people in the room, or at different monitors or, for example, a whiteboard (writing board). Summary of the Invention

[0006] The purpose of this disclosure is to mitigate, alleviate or eliminate one or more of the deficiencies and disadvantages described above in the prior art, and to at least solve the aforementioned problems.

[0007] According to a first aspect, a camera system is provided for generating an image view of a human eye contact, wherein the human has a gaze direction offset from the position of the camera device. The camera system includes a camera device configured to acquire image data of a human, processing circuitry configured to be operatively connected to the camera device and configured to acquire a camera device view of the human from the position of the camera device, and generating a synthetic camera device view of the human seen from a virtual camera device position.

[0008] One advantage of the first aspect is that, regardless of the person's gaze direction (i.e., independently of it), the generated synthetic view of the person will simulate the person's gaze direction, as if the person were looking directly at the camera device, to provide an eye contact experience for the party looking at (seeing) the generated synthetic view of the camera device (i.e., looking towards the synthetic view of the camera device). Therefore, one problem solved is that during a video call, it doesn't appear as if the person is looking elsewhere, but rather, the other party will experience eye contact with that person.

[0009] According to some implementations, the processing circuit is also configured to generate a three-dimensional model of a person from image data acquired by the camera device, acquire real-time two-dimensional image data of the person through the camera device, and generate a synthetic camera device view of the person as seen from the position of the virtual camera device by projecting the real-time two-dimensional image data onto the three-dimensional model based on the person's gaze direction.

[0010] One advantage of the above implementation is that when generating a synthetic view of a person using a camera device, real-time two-dimensional image data (e.g., lip and eye movements) acquired by the camera device is used based on the person's gaze direction (e.g., relative to the camera device).

[0011] According to some implementations, the processing circuit is also configured to determine the gaze direction for the three-dimensional model of a person and to determine the real-time gaze direction of the person.

[0012] One advantage of the above-described implementation is that the generation of the human synthetic camera device view utilizes the human's real-time gaze direction and associates the human's real-time gaze direction with the gaze direction of the three-dimensional model in order to project real-time two-dimensional image data onto the three-dimensional model.

[0013] According to some implementations, the processing circuitry is also configured to set the position of the virtual camera device in the direction of the viewer's gaze.

[0014] One advantage of the above implementation is that when generating a synthetic camera view of a person, the gaze direction provides a good eye contact experience for the person looking at the generated synthetic camera view.

[0015] According to some implementations, the processing circuit is also configured to align the gaze direction of the human's 3D model with the human's real-time gaze direction when real-time 2D image data is projected onto the 3D model.

[0016] One advantage of the above implementation is that the generation of the human synthetic camera device view utilizes the human's real-time gaze direction and aligns the human's real-time gaze direction with the gaze direction of the three-dimensional model in order to project real-time two-dimensional image data onto the three-dimensional model.

[0017] According to some implementations, the processing circuit is also configured to continuously acquire image data through the camera device to improve the three-dimensional model of the person as the person moves their head to different viewpoints (view angles) of the camera device.

[0018] One advantage of the above implementation method is that the three-dimensional model of the human body is improved and updated.

[0019] According to some embodiments, the camera system further includes at least a first camera device and at least a second camera device, which are configured to capture real-time two-dimensional image data of a person to acquire images of the person from at least two different angles.

[0020] One advantage of the above implementation is that, for example, even if a person is looking to the left or right instead of facing forward, at least part of the person's face is always possible to be captured by one of the two camera devices.

[0021] According to some implementations, at least a first camera device and at least a second camera device are configured to capture real-time two-dimensional image data of at least one of a person's eyes and mouth.

[0022] One advantage of the above implementation is that, in order to generate a useful synthetic camera view of a person, at least one moving part of the person's face is required when real-time two-dimensional image data is projected onto a three-dimensional model (i.e., at least part of the face needs to be in motion).

[0023] According to some implementations, the camera system also includes a microphone, and the processing circuitry is further configured to determine spoken language spoken by a person and store image data of facial expressions of the person associated with the spoken language acquired by the camera device to generate predetermined facial expression data.

[0024] One advantage of the above implementation is that it can learn and store facial expressions associated with a particular spoken language.

[0025] According to some embodiments, the camera system also includes a microphone, and the processing circuitry is further configured to determine spoken language spoken by a person and to generate a synthetic facial expression of the person when generating a synthetic camera view of the person based on predetermined facial expression data associated with the spoken language.

[0026] One advantage of the above implementation is that, using predetermined facial expression data associated with the determined spoken language, a synthetic camera view of a person can be generated without fully acquiring the camera view of the person.

[0027] According to a second aspect, a method is provided for generating an image view of a person's gaze contact, wherein the person has a gaze direction deviating from the position of a camera device, the method comprising obtaining a camera device view of the person from the position of the camera device, and generating a synthetic camera device view of the person as seen from a virtual camera device position.

[0028] One advantage of the above is that, regardless of the direction of a person's gaze, the generated synthetic view of the human will simulate the direction of the person's gaze, as if the person were looking directly at the camera device, for the eye contact experience of the person looking at the generated synthetic view of the human.

[0029] According to some implementations, the method further includes the following steps: generating a three-dimensional model of a person using image data acquired by a camera device; acquiring real-time two-dimensional image data of the person using a camera device; and generating a synthetic camera device view of the person as seen from the position of a virtual camera device by projecting the real-time two-dimensional image data onto the three-dimensional model based on the person's gaze direction.

[0030] One advantage of the above implementation is that when generating a synthetic view of a person using a camera device, real-time two-dimensional image data (e.g., lip and eye movements) acquired by the camera device is used based on the person's gaze direction (e.g., relative to the camera device).

[0031] According to some implementations, the method further includes the steps of: determining the gaze direction for a three-dimensional model of a person; and determining the real-time gaze direction of a person.

[0032] One advantage of the above implementation is that the generation of the view of the human synthetic camera device utilizes the human's real-time gaze direction and associates the human's real-time gaze direction with the gaze direction of the three-dimensional model in order to project real-time two-dimensional image data onto the three-dimensional model.

[0033] According to some implementations, the method further includes the steps of: setting the position of the virtual camera device in the direction of the person's gaze; and aligning the gaze direction of the person's three-dimensional model with the person's real-time gaze direction when real-time two-dimensional image data is projected onto the three-dimensional model.

[0034] One advantage of the above implementation is that the generation of the human synthetic camera device view utilizes the human's real-time gaze direction and aligns that direction with the gaze direction of the three-dimensional model in order to project real-time two-dimensional image data onto the three-dimensional model.

[0035] According to a third aspect, a processing circuit program product is provided, comprising a non-transient processing circuit readable medium having a processing circuit program including program instructions on the non-transient processing circuit readable medium, the processing circuit program being loadable into a processing circuit and configured to execute the above-described method when the processing circuit program is run by at least one processing circuit.

[0036] The effects and features of the second and third aspects are largely similar to those described above in conjunction with the first aspect. The implementation methods mentioned in the first aspect are largely compatible with the second and third aspects.

[0037] This disclosure will become apparent from the following detailed description. The specific embodiments and detailed descriptions disclose preferred embodiments of this disclosure by way of illustration only. Those skilled in the art will understand that changes and modifications can be made within the scope of this disclosure based on the guidance in the detailed description.

[0038] Therefore, it should be understood that the content disclosed herein is not limited to the specific components of the described apparatus or the steps of the described method, as such apparatus and methods can be modified. It should also be understood that the terminology used herein is for descriptive purposes only and is not intended to be limiting. It should be noted that when used in the specification and appended claims, the articles “a,” “an,” “the,” and “the” are intended to indicate the presence of one or more elements, unless the context clearly specifies otherwise. Thus, for example, a reference to “a unit” or “the unit” can include several devices, etc. Furthermore, the words “comprising,” “including,” “containing,” and similar wording do not exclude other elements or steps. Attached Figure Description

[0039] The above-described objects, as well as other objects, features, and advantages of this disclosure, will be more fully understood through the following illustrative and non-limiting detailed description of exemplary embodiments of this disclosure, taken in conjunction with the accompanying drawings.

[0040] Figure 1a An exemplary camera system according to this disclosure is shown.

[0041] Figure 1b An exemplary camera system according to this disclosure is shown.

[0042] Figures 2a to 2d Exemplary image data acquired by a camera device according to some embodiments of this disclosure are shown.

[0043] Figures 3a to 3b A three-dimensional model of a person is shown according to some embodiments of this disclosure.

[0044] Figure 4a Exemplary camera devices according to some embodiments of the present disclosure are shown, wherein the camera devices are configured to capture real-time two-dimensional image data of a person.

[0045] Figure 4bImage data of a person captured by a first camera device and a second camera device according to some embodiments of this disclosure are shown.

[0046] Figure 5a Exemplary camera devices according to some embodiments of the present disclosure are shown, the camera devices being configured to capture real-time two-dimensional image data of a person.

[0047] Figure 5b Image data of a person captured by a first camera device and a second camera device according to some embodiments of this disclosure are shown.

[0048] Figure 6a An embodiment of the gaze direction of a person according to some implementations of this disclosure is shown.

[0049] Figure 6b An exemplary synthetic camera device view of a person according to some embodiments of this disclosure is shown.

[0050] Figure 7a Exemplary gaze directions of a person according to some embodiments of this disclosure are shown.

[0051] Figure 7b An exemplary synthetic camera device view of a person according to some embodiments of this disclosure is shown.

[0052] Figure 8 An exemplary synthetic camera device view of a person according to some embodiments of this disclosure is shown.

[0053] Figure 9 An exemplary camera system according to an embodiment of the present disclosure is shown.

[0054] Figure 10 A flowchart of the method steps according to the second aspect of this disclosure is shown.

[0055] Figure 11 A computer program product according to a third aspect of this disclosure is shown. Detailed Implementation

[0056] This disclosure will now be described in conjunction with the accompanying drawings, which illustrate preferred exemplary embodiments of the disclosure. However, the disclosure may be implemented in other ways, and it should not be construed as limiting itself to the embodiments specifically shown. The disclosed embodiments are provided to fully communicate the scope of the disclosure to those skilled in the art.

[0057] Figure 1a An exemplary camera system according to an embodiment of this disclosure is shown.

[0058] A first aspect of this disclosure illustrates a camera system 100 for generating an image view of a human eye contact, wherein the human has a gaze direction offset from the position of the camera device. The camera system 100 includes camera devices 10a, 10b, 10c, and 10d configured to acquire image data of the human.

[0059] According to some implementation methods, a person is a vehicle occupant 1 in a vehicle. Figure 1a An exemplary embodiment is illustrated. As described in the background section, it is particularly difficult to place a camera device where a vehicle occupant is looking, since their eyes should be focused on the road ahead. Vehicle occupant 1 may therefore be looking in a direction through the windshield 11, making it difficult to place the camera device on the windshield 11. Furthermore, the direction of gaze through the windshield 11 changes over time, for example, depending on traffic conditions around the vehicle, making it impossible to place the camera device in all directions of gaze. Vehicle occupant 1 may also be looking around the vehicle through the rear or side windows, or looking at other passengers in mirrors, rearview camera displays, etc. Figure 1a In the embodiment shown, the gaze direction of vehicle occupant 1 is towards vehicle 21 in front of the vehicle.

[0060] According to some embodiments, the camera system 100 also includes a display 40 configured to display an image view of the other party during a video call. According to some embodiments, the display 40 is a heads-up display. Figure 1a In the embodiment shown, the display 40 is a head-up display that is projected onto the windshield 11 in front of the vehicle occupant 1.

[0061] According to some embodiments, camera devices 10a, 10b, and 10c are camera devices installed at predetermined camera device positions. Figure 1a In the illustrated embodiment, camera devices 10a, 10b, and 10c are camera devices installed at predetermined camera locations inside the vehicle. In one example, existing camera devices at different camera locations within the vehicle are used as camera devices for camera system 100.

[0062] According to some embodiments, the camera device is the camera device 10d of the portable electronic device 400. In one example, the portable electronic device 400 is either a smartphone or a tablet operably connected to the camera system 100 via either a wired communication network or a wireless communication network 50. In one example, the portable electronic device 400 is housed in a bracket located on the dashboard of a vehicle, with the camera device 10d of the portable electronic device 400 facing a person (i.e., vehicle occupant 1).

[0063] According to some embodiments, the camera positions of camera devices 10a, 10b, 10c, and 10d are predetermined camera device positions relative to the position of a person. According to some embodiments, when acquiring image data of a person through multiple camera devices 10a, 10b, 10c, and 10d, position information regarding the positions of the multiple camera devices 10a, 10b, 10c, and 10d relative to the position of the person is used.

[0064] In the embodiment shown in FIG1, exemplary camera devices 10a, 10b, and 10c are positioned around the windshield 11 at camera device locations, and one camera device location is defined by the location of camera device 10d of the portable electronic device 400. In the embodiment shown in FIG1, the gaze direction of the vehicle occupant 11 is therefore deviated from any camera device location.

[0065] The camera system 100 also includes processing circuits 102a, 102b, and 102c, which are configured to be operatively connected to the camera devices 10a, 10b, 10c, and 10d.

[0066] According to some embodiments, processing circuit 102a is the processing circuit of an on-board computer. According to some embodiments, as shown in FIG1, processing circuits 102b and 102c are included in electronic devices 400 and 500 that can be connected to camera system 100 via wireless communication network 50.

[0067] According to some embodiments, the camera system 100 also includes memories 101a, 101b, and 101c configured to store data. According to some embodiments, memory 101a is the memory of an onboard computer. According to some embodiments, as shown in FIG1, memories 101b and 101c are included in electronic devices 400 and 500 that can be connected to the camera system 100 via a wireless communication network 50.

[0068] According to some implementation methods, as shown in Figure 1 and... Figure 9 As shown, the wireless communication network 50 is a standardized wireless local area network, such as WLAN or Bluetooth. TMThis can be a ZigBee, ultra-wideband, RFID, or similar network. In one example, the wireless communication network 50 is a standardized wireless wide area network, such as GSM, Extended GSM, GPRS, Enhanced Data GSM Environment (EDGE), WCDMA, LTE, Narrowband Radio Access, 5G, WiMAX, or UMB, or a similar network. According to some implementations, the wireless communication network 50 can also be a combination of a local area network (LAN) and a wide area network (WAN). According to some implementations, the wireless communication network 50 is defined by a public Internet Protocol (IP).

[0069] Processing circuits 102a, 102b, and 102c are configured to acquire a view of a person from the camera device's location and generate a composite view of the person from the virtual camera device's location. Figure 1a In the illustrated embodiment, the location of the virtual camera device is indicated by a dashed circle “VCP”, which is in the direction of the vehicle occupant 1’s gaze. In one example, such as during a video call, the vehicle occupant is looking at an image view presented on the other side of the display 40, and the direction of the vehicle occupant 1’s gaze is therefore at the display 40, which thus becomes the location of the virtual camera device.

[0070] One advantage of the above implementation is that, regardless of the person's gaze direction, the generated synthetic view of the person from the camera device will simulate the person's gaze direction, as if the person were looking directly at the camera device, to provide an eye contact experience for the party looking at the generated synthetic view of the camera device. Therefore, one problem solved is that during a video call, it does not appear as if the person is looking elsewhere, but rather, the other party will experience eye contact with that person.

[0071] According to some embodiments, the synthetic camera device view of a person is an image view of the person's eye contact. According to some embodiments, the synthetic camera device view of a person as seen from the location of a virtual camera device is used to generate a video stream for making a video call with another party.

[0072] According to some embodiments, a video stream from another party is presented to the person at a display 40 in front of them. According to some embodiments, a video stream from another party is projected from a head-up display 40 onto a windshield 11 in front of the vehicle occupant 1 and presented to the vehicle occupant 1.

[0073] According to some implementations, the processing circuits 102a, 102b, and 102c are also configured to generate a three-dimensional model of a person using image data acquired by the camera devices 10a, 10b, 10c, and 10d.

[0074] According to some embodiments, the processing circuits 102a, 102b, and 102c are also configured to process the image data acquired by the camera devices 10a, 10b, 10c, and 10d through photogrammetric processing to generate a three-dimensional model of the person.

[0075] According to some implementations, the processing circuits 102a, 102b, 102c are also configured to extract three-dimensional measurement dimensions from the two-dimensional image data acquired by the camera devices 10a, 10b, 10c, 10d to generate a three-dimensional model of a person.

[0076] Figures 2a to 2d Exemplary image data acquired by camera devices 10a, 10b, 10c, and 10d according to some embodiments of the present disclosure are shown. Figures 2a to 2d In the embodiment shown, the person follows as Figures 2a to 2d The illustrations depict the head moving in different directions, thus acquiring multiple image data of the person from multiple directions.

[0077] According to some implementation methods, the three-dimensional model of the person is stored in memories 101a, 101b, and 101c.

[0078] According to some embodiments, a three-dimensional model of a person is created using multiple images from different camera devices 10a, 10b, 10c, and 10d located at different camera device positions. According to some embodiments, such as... Figure 3a and Figure 3b As shown, the three-dimensional model of a person is the three-dimensional geometry of a person's face. According to some embodiments, processing circuits 102a, 102b, and 102c are also configured to generate the three-dimensional model of the person using image recognition processing (method).

[0079] In one example, when the camera system 100 for generating a 3D model of a person is activated, the person (e.g., vehicle occupant 1) is prompted to move their head in different directions during the learning phase. In another example, when the camera system 100 is used while the vehicle is in motion, the person (e.g., vehicle occupant 1) moves their head in different directions, and the camera system 100 automatically generates a 3D model of the person during a predetermined time period or when a predetermined amount of image data is acquired.

[0080] According to some embodiments, the three-dimensional model data is predetermined three-dimensional model data of a person associated with facial recognition data and stored in memories 101a, 101b, 101. According to some embodiments, the processing circuits 102a, 102b, 102c of the camera system 100 are also configured to acquire image data of a person and compare facial recognition data acquired from the image data of the person with facial recognition data of the stored three-dimensional model data in memories 101a, 101b, 101c to utilize the existing (pre-existing) predetermined three-dimensional model of the person.

[0081] According to some embodiments, a three-dimensional model of a person is determined by acquiring image data of the moving parts of the human face. According to some embodiments, the moving parts of the human face are any movement of at least one of the following: lips, eyes, eyebrows, jaw, cheeks, nose, ears, wrinkles, or dimples.

[0082] According to some implementations, the processing circuits 102a, 102b, and 102c are also configured to acquire real-time two-dimensional image data of a person through the camera devices 10a, 10b, 10c, and 10d, and to generate a synthetic camera device view of the person as seen from the position of the virtual camera device by projecting the real-time two-dimensional image data onto a three-dimensional model based on the person's gaze direction.

[0083] exist Figure 1a In the illustrated embodiment, the camera device 10a acquires real-time two-dimensional image data of a person. Figure 1a In the example shown, the person is vehicle occupant 1, and vehicle occupant 1 is driving the vehicle, with their gaze directed towards vehicle 21 in front of them. In this embodiment, the real-time two-dimensional image data of vehicle occupant 1 acquired by camera device 10a is similar to... Figure 6a The image shown. Therefore, in this embodiment, the vehicle occupant is looking in a direction offset from the camera device 10a. According to some embodiments, processing circuits 102a, 102b, 102c are configured to project real-time two-dimensional image data onto a three-dimensional model based on the person's gaze direction. In this embodiment, the real-time two-dimensional image data of the vehicle occupant is similar to... Figure 6a The two-dimensional image data shown has a gaze direction offset from the camera device position 10a and contains knowledge of the human's three-dimensional model. Real-time two-dimensional image data can be projected onto the three-dimensional model based on the human's gaze direction. The projection result is a synthetic camera device view of the human as seen from the virtual camera device position, such as... Figure 6b A view of an exemplary synthetic camera device is shown. Figure 1a In the illustrated embodiment, the location of the virtual camera device is indicated by the dashed circle "VCP".

[0084] For illustrative purposes, in Figure 7a Another example of real-time 2D image data is shown, which has a gaze direction offset from the camera device position of camera device 10a and incorporates knowledge of a human 3D model, allowing the real-time 2D image data to be projected onto the 3D model based on the human's gaze direction. In this example, the result of the projection is a synthetic camera device view of the human as seen from the virtual camera device position, such as... Figure 7b The example shown is a synthetic camera device view. In other words, the synthetic camera device view of a person seen from the location of the virtual camera device is substantially the same, regardless of the person's gaze direction in the real-time two-dimensional image data. Therefore, the result is a continuous eye contact experience for the party looking at the generated synthetic camera device view.

[0085] For illustrative purposes, Figure 6b and 7b The exemplary diagram in the figure shows, as follows Figure 6a and 6b The image shows an overlay of a synthetic camera view onto real-time 2D image data. This is to visualize the effect of generating a synthetic camera view of a person as seen from the virtual camera's position when real-time 2D image data is projected onto a 3D model based on the person's gaze direction. This allows for a continuous eye contact experience through the generated synthetic camera view, regardless of the person's gaze direction.

[0086] Figure 8 An exemplary synthetic imaging device view of a person according to some embodiments of this disclosure is shown. Figure 8 In the exemplary illustration, when real-time two-dimensional image data is projected onto a three-dimensional model, at least the moving parts of a human face are processed. According to some implementations, when real-time two-dimensional image data is projected onto a three-dimensional model, image processing is used to smooth the boundary between the real-time two-dimensional image data and the three-dimensional model, so that the projection of the real-time two-dimensional image data onto the three-dimensional model is seamless (connected).

[0087] One advantage of the above-described embodiments is that, as Figure 8 As shown, regardless of the direction of a person's gaze, the generated synthetic view of the human will simulate the direction of the person's gaze, as if the person were looking directly at the camera device for the eye contact experience of looking at the side of the generated synthetic view of the human.

[0088] Another advantage of the above implementation is that when generating a synthetic view of a person using a camera device, real-time two-dimensional image data acquired by the camera device, such as the movement of at least the lips and eyes, is used based on the person's gaze direction (e.g., relative to the camera device).

[0089] According to some embodiments, processing circuits 102a, 102b, and 102c are further configured to determine the gaze direction for the three-dimensional model of the person and to determine the real-time gaze direction of the person. According to some embodiments, the gaze direction is determined by at least one of the person's eyes, nose, ears, or mouth.

[0090] According to some embodiments, the gaze direction used for the 3D model is predetermined and associated with the 3D model of a person. According to some embodiments, the 3D model and the gaze direction used for the 3D model are stored as 3D model data in memories 101a, 101b, and 101c.

[0091] According to some embodiments, the real-time gaze direction of a person is determined based on real-time two-dimensional image data of the person acquired by camera devices 10a, 10b, 10c, and 10d. According to some embodiments, the real-time gaze direction of the person is continuously determined and associated with the real-time two-dimensional image data of the person.

[0092] One advantage of the above implementation is that the generation of the view of the human synthetic camera device utilizes the human's real-time gaze direction and associates the human's real-time gaze direction with the gaze direction of the three-dimensional model in order to project real-time two-dimensional image data onto the three-dimensional model.

[0093] According to some embodiments, the processing circuits 102a, 102b, 102c are also configured to set the position of the virtual camera device in the direction of the person's gaze.

[0094] One advantage of the above implementation is that when generating a synthetic camera view of a person, the gaze direction provides a good eye contact experience for the person looking at the generated synthetic camera view.

[0095] According to some embodiments, the processing circuits 102a, 102b, and 102c are further configured to set the position of the virtual camera device in the direction of human gaze during a first time period, and to set the position of the virtual camera device away from the direction of human gaze during a second time period.

[0096] One advantage of the above-described implementation is that, when generating a synthetic camera view of a person, the gaze direction provides a good eye contact experience for the person looking at the generated synthetic camera view during a first time period, followed by a gaze that deviates from that eye contact, thereby providing the person looking at the generated synthetic camera view with a natural experience and a lower gaze intensity.

[0097] According to some implementations, when real-time two-dimensional image data is projected onto a three-dimensional model based on the human's gaze direction, the human's gaze direction in the three-dimensional model and the human's real-time gaze direction are used to generate a synthetic camera view of the human as seen from the location of the virtual camera device.

[0098] According to some implementations, the processing circuits 102a, 102b, and 102c are also configured to align the gaze direction of the human's three-dimensional model with the human's real-time gaze direction when real-time two-dimensional image data is projected onto the three-dimensional model.

[0099] One advantage of the above implementation is that the generation of the human synthetic camera device view utilizes the human's real-time gaze direction and aligns the human's real-time gaze direction with the gaze direction of the three-dimensional model in order to project real-time two-dimensional image data onto the three-dimensional model.

[0100] According to some implementations, the processing circuits 102a, 102b, and 102c are also configured to continuously acquire image data through the camera devices 10a, 10b, 10c, and 10d to improve the three-dimensional model of the person when the person moves their head at different view angles of the camera devices 10a, 10b, 10c, and 10d.

[0101] One advantage of the above implementation method is that the three-dimensional model of the human body is improved and updated to the latest version.

[0102] According to some embodiments, the camera system 100 further includes at least a first camera device 10a and at least a second camera device 10b, which are configured to capture real-time two-dimensional image data of a person from at least two different angles to obtain an image of the person.

[0103] According to some implementations, the first camera device 10a and the second camera device 10b capture real-time two-dimensional image data of a person at the same time or within a predetermined time period.

[0104] According to some implementations, image data is acquired by a first camera device 10a and a second camera device 10b at the same time or within a predetermined time period, and processed to generate a three-dimensional model of a person from the image data acquired by the camera devices 10a, 10b, 10c, and 10d.

[0105] According to some embodiments, the processing circuits 102a, 102b, and 102c are further configured to acquire real-time two-dimensional image data of a person at the same time or within a predetermined time period by the first camera device 10a and the second camera device 10b, and to generate a synthetic camera device view of the person as seen from the position of the virtual camera device by projecting the real-time two-dimensional image data onto a three-dimensional model based on the person's gaze direction.

[0106] The advantage of configuring at least the first camera device 10a and at least the second camera device 10b to capture real-time two-dimensional image data of a person to obtain images of the person from at least two different angles is that, in the case of a vehicle occupant, the occupant may not always be facing one direction (usually the opposite of forward), but may be facing around the vehicle, such as through a rear or side window, or looking at other passengers in a mirror, a rearview camera display, etc. Therefore, the other camera devices can cover multiple angles, preferably 360 degrees around the vehicle occupant.

[0107] Figure 4a and 5b An exemplary camera system 100 is shown, comprising at least a first camera device 10a and at least a second camera device 10b. Figure 4b In the exemplary illustration, dots in the face represent image data acquired by the camera device 10a. Figure 4b In the exemplary illustration, the lines in the face represent image data acquired by camera device 10b. In the exemplary illustration in 4a, the person's gaze direction is between the camera device positions of camera device 10a and camera device 10b. In this embodiment, as... Figure 4b As shown, the image data of the person is acquired by both camera device 10a and camera device 10b.

[0108] exist Figure 5b In the exemplary illustration, dots in the face represent image data acquired by the camera device 10a. Figure 5b In the exemplary illustration, the lines in the face represent image data acquired by camera device 10b. In the exemplary illustration in 5a, the person's gaze is directed to the right, and more towards the camera device position of camera device 10b, and less towards the camera device position of camera device 10a. In this embodiment, as... Figure 5b As shown, the image data of the person is acquired by both camera device 10a and camera device 10b, and therefore the image data acquired by camera device 10a is less than the image data acquired by camera device 10b.

[0109] According to some embodiments, multiple camera devices 10a, 10b, 10c, and 10d are used to continuously acquire image data. According to some embodiments, multiple image data of a person are acquired from multiple directions by the multiple camera devices 10a, 10b, 10c, and 10d, and processed to generate at least one of real-time two-dimensional image data of the person and a three-dimensional model of the person.

[0110] One advantage of the above implementation is that even if a person is looking, for example, in a direction to the left or right instead of forward, there is a possibility that at least a part of the person's face will always be captured by one of the two camera devices.

[0111] Figure 1b An exemplary camera system according to an embodiment of the present disclosure is shown. In this embodiment, two camera devices 10a and 10b are used to generate at least one of real-time two-dimensional image data of a person and a three-dimensional model of the person.

[0112] According to some embodiments, at least a first gaze direction in real-time two-dimensional image data of a person acquired by a first camera device 10a and at least a second gaze direction in real-time two-dimensional image data of a person acquired by a second camera device 10b are processed to improve the generation of a synthetic camera device view of a person as seen from the virtual camera device position when the processed real-time two-dimensional image data acquired by the first camera device 10a and the second camera device 10b are projected onto a three-dimensional model based on the person's gaze direction. Figure 1b In the exemplary illustration, the head-up display 40 displays an image view of the other party during a video call.

[0113] According to some embodiments, a synthetic camera device view of a person as seen from the position of a virtual camera device is continuously determined. According to some embodiments, the synthetic camera device view of a person as seen from the position of the virtual camera device is continuously determined and generated based on changes in the person's gaze direction. Figure 1b In the exemplary illustration, the virtual camera device position "VCP," as indicated by the dashed circle, moves continuously according to the direction of the viewer's gaze. Figure 1b In one embodiment, the virtual camera location "VCP" is primarily located at a point on the windshield 11, but it can also be located at different locations.

[0114] According to some embodiments, at least a first camera device 10a and at least a second camera device 10b are configured to capture real-time two-dimensional image data of at least one of a person's eyes and mouth.

[0115] According to some implementations, multiple camera devices 10a, 10b, 10c, and 10d are used to capture real-time two-dimensional image data of at least one of a person's eyes and mouth.

[0116] According to some implementations, the processing circuits 102a, 102b, and 102c are also configured to generate a synthetic camera device view of the person as seen from the location of the virtual camera device by projecting real-time two-dimensional image data of at least one of the person's eyes and mouth onto a three-dimensional model based on the person's gaze direction.

[0117] In one example, two or more camera devices 10a, 10b, 10c, and 10d are placed so that they together capture all parts of the user's face that move and change during a normal call. In this example, the lips and cheeks need to be covered, but the ears can be excluded.

[0118] One advantage of the above implementation is that, in order to generate a useful synthetic camera view of a person, at least the moving part of the person's face is required when real-time two-dimensional image data is projected onto a three-dimensional model.

[0119] According to some embodiments, the camera system 100 also includes microphones 30a, 30b, 30c, and 30d, and processing circuits 102a, 102b, and 102c are further configured to determine spoken language spoken by a person and store image data of facial expressions of a person associated with spoken language acquired by the camera devices 10a, 10b, 10c, and 10d to generate predetermined facial expression data.

[0120] According to some implementations, the processing circuits 102a, 102b, and 102c are also configured to determine a person's emotion based on facial expressions acquired by the camera devices 10a, 10b, 10c, and 10d and the sound of spoken language acquired by the microphones 30a, 30b, 30c, and 30d, and to store the person's emotion data associated with the sound of spoken language and facial expressions to generate predetermined facial expression data.

[0121] According to some implementations, microphones 30a, 30b, 30c, and 30d are microphones installed at predetermined locations. Figure 1a In the illustrated embodiment, microphones 30a, 30b, and 30c are microphones installed at predetermined locations within the vehicle. In one example, existing microphones at different locations within the vehicle are used as microphones for the camera system 100. According to some embodiments, the microphone is microphone 30d of a portable electronic device 400. In one example, the portable electronic device 400 is either a smartphone or a tablet computer operably connected to the camera system 100 via a wireless communication network 50.

[0122] According to some embodiments, the processing circuits 102a, 102b, and 102c are also configured to perform speech-to-text processing to associate spoken language with text and store image data of a person’s facial expressions associated with spoken language and text acquired by the camera devices 10a, 10b, 10c, and 10d to generate predetermined facial expression data.

[0123] In one example, when the camera system 100 for generating predetermined facial expression data is activated, a person (e.g., vehicle occupant 1) is prompted to speak spoken language and utter different words or letters during the learning phase. In another example, vehicle occupant 1 converses with other passengers in the vehicle during normal conversation and learns multiple words and letters associated with different facial expressions through the camera system 100. During a predetermined time period or when a predetermined number of words or letters are acquired, the camera system 100 automatically generates predetermined facial expression data associated with the person.

[0124] One advantage of the above implementation is that facial expressions associated with a particular spoken language can be learned and stored.

[0125] According to some embodiments, the camera system 100 also includes microphones 30a, 30b, 30c, and 30d, and processing circuits 102a, 102b, and 102c are further configured to determine spoken language spoken by a person and generate synthetic facial expressions of a person when a synthetic camera device view of a person is generated based on predetermined facial expression data associated with the spoken language.

[0126] One advantage of the above implementation is that a synthetic camera view of a person can be generated using predetermined facial expression data associated with a defined spoken language, without having to fully acquire the camera view of the person.

[0127] According to some implementations, when it is determined that no camera device 10a, 10b, 10c, 10d acquires image data of a person, but the person's spoken language is determined, the processing circuits 102a, 102b, 102c are also configured to generate a synthetic facial expression of a person based on the spoken language and based on predetermined facial expression data associated with the spoken language.

[0128] In one example, even if a person is looking in a direction from which no image data of the person's face can be acquired by camera devices 10a, 10b, 10c, and 10d, the synthetic camera device view of the person can still simulate known facial expressions associated with spoken language. In another example, a vehicle occupant may be reversing and turning their face towards the rear window, but if camera devices 10a, 10b, 10c, and 10d are positioned in front of the occupant, the synthetic camera device view of the person in this situation can still simulate known facial expressions associated with spoken language.

[0129] In another example, due to reasons such as strong light, lack of light, or light pollution of the camera device, the camera devices 10a, 10b, 10c, and 10d have difficulty acquiring image data of people.

[0130] According to some implementations, camera devices 10a, 10b, 10c, and 10d are configured to be turned off, and based on the determination that camera devices 10a, 10b, 10c, and 10d are turned off, when a synthetic facial expression of a person is generated based on predetermined facial expression data associated with spoken language, a synthetic facial expression of the person is generated. In one example, a person may not want to be in a video call with their current appearance; instead, predetermined facial expression data associated with spoken language is used instead of that current appearance.

[0131] Another example is in an office environment, such as when multiple monitors are used during a video call but only one camera is used to capture a person, it is difficult to place the camera where the person is looking. Therefore, when the first person is looking at the first monitor equipped with the camera, the video call experience may be sufficient for the second person, but when the first person is looking at the second monitor, the second person may experience, for example, looking at the first person's cheek rather than their eyes.

[0132] Figure 9 An exemplary camera system 100 according to an embodiment of the present disclosure is shown.

[0133] According to some implementations, the person is a video conference member 2 in a conference room. Exemplary implementations are as follows: Figure 9 As shown in the image.

[0134] As described in the background section, placing the camera device where video conference participant 2 is looking can be difficult. It might be more natural for participant 2 in the meeting room to not always be looking at the camera device, but rather at other people in the meeting room, or at a different monitor, or at a whiteboard. Furthermore, the direction of gaze will change over time, depending on where participant 2 is looking, for example, at different people in the meeting room, and therefore it is impossible to place the camera device in all directions of gaze. Figure 9 In the embodiment shown, the video conference member 2 is looking at another person in the conference room.

[0135] According to some embodiments, the camera system 100 also includes a display 40 configured to display an image view of the other party during a video call. According to some implementations, the display 40 is a computer monitor. Figure 9 In the embodiment shown, display 40 is a large computer monitor on the wall of a conference room.

[0136] In one example, video conference participant 2 is looking at different objects displayed on the screen area of ​​monitor 40 (e.g., a computer monitor). Figure 9As shown, even if a video conference member 2 is looking at the monitor 40, for example during a video call, looking at an image view of the other side presented in the first part of the display area of ​​the monitor 40 or in the second part of the display area of ​​the monitor 40, the video conference member 2 will not look directly at the camera device 10a placed above the monitor 40.

[0137] However, by utilizing the various aspects and implementations of the camera system 100 previously described in this disclosure, one effect is that during a video call, it will not appear as if video conference member 2 is looking elsewhere, such as at other people in the conference room, at a different monitor or whiteboard, or at a different part of the display area of ​​monitor 40, but rather the other party will experience eye contact with video conference member 2.

[0138] A second aspect of this disclosure illustrates a method for generating an image view of a person's gaze contact, wherein the person has a gaze direction that is offset from the position of the camera device. Figure 10 A flowchart of the method steps according to the second aspect of this disclosure is shown.

[0139] The method includes step S1: obtaining a camera device view of a person from the camera device location; and step S8a: generating a synthetic camera device view of a person seen from the virtual camera device location.

[0140] One advantage of this is that, regardless of the direction of a person's gaze, the generated synthetic view of the human device will simulate the direction of the person's gaze, as if the person were looking directly at the camera device, to provide a visual experience of the person looking at the generated synthetic view of the human device.

[0141] According to some implementations, the method further includes step S2: generating a three-dimensional model of a person using image data acquired by camera devices 10a, 10b, 10c, and 10d; step S4: acquiring real-time two-dimensional image data of the person using camera devices 10a, 10b, 10c, and 10d; and step S8b: generating a synthetic camera device view of the person as seen from the position of the virtual camera device by projecting the real-time two-dimensional image data onto the three-dimensional model based on the person's gaze direction.

[0142] One advantage of the above implementation is that when generating a synthetic view of a person using a camera device, real-time two-dimensional image data (e.g., lip and eye movements) acquired by the camera device is used based on the person's gaze direction (e.g., relative to the camera device).

[0143] According to some implementations, the method further includes step S3: determining the gaze direction of the three-dimensional model of the person; and step S5: determining the real-time gaze direction of the person.

[0144] One advantage of the above implementation is that the generation of the view of the human synthetic camera device utilizes the human's real-time gaze direction and associates the human's real-time gaze direction with the gaze direction of the three-dimensional model in order to project real-time two-dimensional image data onto the three-dimensional model.

[0145] According to some implementations, the method further includes step S6: setting the position of the virtual camera device to be in the direction of the person's gaze; and step S7: aligning the gaze direction of the person's three-dimensional model with the person's real-time gaze direction when real-time two-dimensional image data is projected onto the three-dimensional model.

[0146] One advantage of the above implementation is that the generation of the human synthetic camera device view utilizes the human's real-time gaze direction and aligns the human's real-time gaze direction with the gaze direction of the three-dimensional model in order to project real-time two-dimensional image data onto the three-dimensional model.

[0147] Figure 11 A computer program product according to a third aspect of this disclosure is shown. The third aspect of this disclosure shows a processing circuit program product comprising a non-transient processing circuit readable medium having a processing circuit program including program instructions thereon, the processing circuit program being loadable into processing circuits 102a, 102b, 102c and configured to execute a method when the processing circuit program is run by at least one processing circuit 102a, 102b, 102c.

[0148] Those skilled in the art will understand that this disclosure is not limited to the preferred embodiments described above. They will also recognize that modifications and variations are possible within the scope of the appended claims. Furthermore, by studying the drawings, the disclosure, and the appended claims, those skilled in the art can understand and implement variations of the disclosed embodiments when practicing the claimed disclosure.

Claims

1. A camera system (100) for generating an image view of a person's gaze contact, wherein the person has a gaze direction offset from the position of the camera device, the camera system (100) comprising: Camera devices (10a, 10b, 10c, 10d) are configured to acquire image data of the person; Processing circuits (102a, 102b, 102c) are configured to be operably connected to the imaging device (10a, 10b, 10c, 10d) and are configured to: - Obtain a view of the person from the camera device's location; - Determine the spoken language spoken by the person in question; - Store image data of the person's facial expressions associated with the spoken language acquired by the camera devices (10a, 10b, 10c, 10d) to generate predetermined facial expression data; - Generate a synthetic camera view of a person as seen from the location of a virtual camera device; as well as - When a synthetic camera view of the person is generated based on predetermined facial expression data associated with the spoken language, a synthetic facial expression of the person is generated.

2. The camera system (100) according to claim 1, wherein, The processing circuits (102a, 102b, 102c) are also configured to: - Generate a three-dimensional model of the person using image data acquired by the camera devices (10a, 10b, 10c, 10d); - Real-time two-dimensional image data of the person is acquired through the camera devices (10a, 10b, 10c, 10d); and By projecting the real-time two-dimensional image data onto the three-dimensional model based on the person's gaze direction, a synthetic camera device view of the person as seen from the position of the virtual camera device is generated.

3. The camera system (100) according to claim 2, wherein, The processing circuits (102a, 102b, 102c) are also configured to: - Determine the gaze direction for the three-dimensional model of the person; and - Determine the person's real-time gaze direction.

4. The camera system (100) according to claim 1, wherein, The processing circuits (102a, 102b, 102c) are also configured to: - Set the position of the virtual camera device in the direction of the person's gaze.

5. The camera system (100) according to claim 2, wherein, The processing circuits (102a, 102b, 102c) are also configured to: - When the real-time two-dimensional image data is projected onto the three-dimensional model, the gaze direction of the three-dimensional model of the person is aligned with the real-time gaze direction of the person.

6. The camera system (100) according to claim 2, wherein, The processing circuits (102a, 102b, 102c) are also configured to continuously acquire image data through the camera devices (10a, 10b, 10c, 10d) when the person moves their head at different perspectives of the camera devices (10a, 10b, 10c, 10d) to improve the three-dimensional model of the person.

7. The camera system (100) according to any one of claims 1-6, wherein, The camera system (100) further includes at least a first camera device (10a) and at least a second camera device (10b), which are configured to capture real-time two-dimensional image data of the person to obtain images of the person from at least two different angles.

8. The camera system (100) according to claim 7, wherein, The at least first camera device (10a) and the at least second camera device (10b) are configured to capture real-time two-dimensional image data of at least one of the person's eyes and mouth.

9. A method for generating an image view of a human eye contact, wherein the human has a gaze direction offset from the position of a camera device, the method comprising: - (S1) A view of the person from the camera device location; - Determine the spoken language spoken by the person in question; - Store image data of the person's facial expressions associated with the spoken language acquired by the camera devices (10a, 10b, 10c, 10d) to generate predetermined facial expression data; - (S8a) Generate a synthetic camera device view of the person as seen from the location of the virtual camera device; as well as - When a synthetic camera view of the person is generated based on predetermined facial expression data associated with the spoken language, a synthetic facial expression of the person is generated.

10. The method according to claim 9, further comprising: - (S2) Generate a three-dimensional model of the person using image data acquired by the camera devices (10a, 10b, 10c, 10d); - (S4) Acquire real-time two-dimensional image data of the person through the camera devices (10a, 10b, 10c, 10d); and - (S8b) By projecting the real-time two-dimensional image data onto the three-dimensional model based on the person's gaze direction, a synthetic camera device view of the person as seen from the location of the virtual camera device is generated.

11. The method of claim 10, further comprising: - (S3) Determine the gaze direction for the three-dimensional model of the person; as well as - (S5) Determine the real-time gaze direction of the person.

12. The method according to claim 10 or 11, further comprising: - (S6) Set the position of the virtual camera device in the direction of the person's gaze; as well as - (S7) When the real-time two-dimensional image data is projected onto the three-dimensional model, the gaze direction of the three-dimensional model of the person is aligned with the real-time gaze direction of the person.

13. A processing circuit program product (500) comprising a non-transient processing circuit readable medium having a processing circuit program including program instructions on the non-transient processing circuit readable medium, the processing circuit program being loadable into a processing circuit (102a, 102b, 102c) and configured to perform the method according to any one of claims 9-12 when the processing circuit program is run by at least one processing circuit (102a, 102b, 102c).

Images