Stereoscopic image data synthesis method, device, electronic device and storage medium

Abstract

The present application provides a method, device, electronic device and storage medium for synthesizing stereoscopic image data, which belongs to the technical field of image processing. The method for synthesizing stereoscopic image data includes: obtaining facial image data of a target object in a viewer image; obtaining the coordinates of the center points of the two eyes of the target object in the viewer image according to the facial image data; determining the compensation value of the coordinates of the two eyes relative to the preset initial center coordinates; determining the compensation weights of the left eye image data and the right eye image data with parallax used to synthesize the stereoscopic image data according to the compensation value; synthesizing according to the left eye graphic data, the right eye image data and the respective compensation weights to obtain the stereoscopic image data to be displayed. The stereoscopic image data to be displayed obtained by this solution focuses on the two eyes of the target object, which can provide the best stereoscopic image data display effect for the target object, and realizes the effect of switching the focus object as needed.

Description

Technical Field

[0001] The present application relates to the technical field of image processing, and in particular to a method, device, electronic device and storage medium for synthesizing stereoscopic image data. Background Art

[0002] Because naked-eye 3D displays can present three-dimensional depth, they have a more realistic visual immersion. The current mainstream naked-eye 3D displays use slit grating or cylindrical lens technology, which has the problem of discontinuous viewing areas. The images seen in some areas have ghosting or reverse parallax, causing dizziness or discomfort. The industry uses human eye tracking to adjust the phase of the display's synthesized 3D interlaced image in real time according to the position of the person captured by the camera, so that the viewer can see a stereoscopic synthetic image with a correct parallax relationship no matter where he moves. However, existing human eye tracking is only suitable for single-person capture and tracking. When there are multiple people watching the stereoscopic synthetic image at the same time (such as application scenarios such as remote conferencing), the camera cannot determine who should be tracked at the moment, and based on the judgment of global static features (such as the primary and secondary relationship of the seat layout), it is impossible to switch the focus tracking object as needed. Summary of the invention

[0003] The present application provides a stereoscopic image data synthesis method, device, electronic device and storage medium to solve the problem that when multiple people watch a stereoscopic synthetic image at the same time, the existing human eye tracking technology cannot determine who should be tracked, and the judgment based on global static features cannot switch the focus tracking object as needed.

[0004] In a first aspect, the present application provides a method for synthesizing stereoscopic image data, which is applied to an electronic device comprising an image acquisition unit, wherein the image acquisition unit is used to acquire an image of a viewer viewing a stereoscopic synthesized image, the method comprising: acquiring an azimuth angle and a pitch angle of a target object relative to a preset position; obtaining a horizontal coordinate of a face of the target object in the viewer image based on the azimuth angle and a horizontal resolution and a horizontal field of view angle of the image acquisition unit acquired in advance; obtaining a vertical coordinate of the face of the target object in the viewer image based on the pitch angle and a vertical resolution and a vertical field of view angle of the image acquisition unit acquired in advance; and obtaining a vertical coordinate of the face of the target object in the viewer image based on the target object's face. The method comprises the following steps: obtaining the face image data of a target object from the viewer image by using the horizontal coordinate and the vertical coordinate of a face in the viewer image; obtaining the eye center point coordinates of the center points of the eyes of the target object in the viewer image according to the face image data; determining the compensation value of the eye center point coordinates relative to the preset initial center coordinates; determining the compensation weights of the left eye image data and the right eye image data with parallax for synthesizing the stereoscopic image data according to the compensation value; synthesizing according to the left eye graphic data, the right eye image data and the compensation weights of the left eye image data and the right eye image data to obtain the stereoscopic image data to be displayed.

[0005] In an embodiment of the present application, by acquiring the azimuth and pitch angles of the target object relative to a preset position, and then based on the azimuth and the horizontal resolution and lateral field of view of the image acquisition unit acquired in advance, the horizontal coordinate of the target object's face in the viewer's image can be accurately obtained, and then based on the pitch angle and the vertical resolution and longitudinal field of view of the image acquisition unit acquired in advance, the vertical coordinate of the target object's face in the viewer's image can be accurately obtained. Through the precise position coordinates of the target face, the face of the target object can be accurately located from the viewer's image, effectively preventing the occurrence of incorrect face positioning, improving the accuracy of face positioning, and ensuring the accuracy of the acquired face image data. At the same time, by acquiring the center coordinates of both eyes of the target object in the viewer's image, and then determining the compensation value of the center coordinates of the two eyes relative to the preset initial center coordinates, the compensation weights of the left-eye image data and the right-eye image data are determined according to the compensation value, so that the stereoscopic image data to be displayed is focused on the two eyes of the target object by synthesizing according to the left-eye graphic data, the right-eye image data and the respective compensation weights of the left-eye image data and the right-eye image data. Then, when displaying the stereoscopic image data to be displayed, the best stereoscopic image data display effect can be provided for the target object, and the effect of switching the focus object as needed is achieved.

[0006] In combination with the technical solution provided in the first aspect above, in some possible implementations, if the position of the image acquisition unit is set not to coincide with the preset position, the method further includes: obtaining a first distance of the target object relative to the preset position, and a second distance between the preset position and the image acquisition unit; obtaining the longitudinal coordinate of the face of the target object in the viewer image based on the pitch angle and the vertical resolution and longitudinal field of view of the image acquisition unit obtained in advance, including: obtaining the longitudinal coordinate of the face of the target object in the viewer image based on the pitch angle, the first distance, the second distance and the vertical resolution and longitudinal field of view of the image acquisition unit obtained in advance.

[0007] In an embodiment of the present application, when the position of the image acquisition unit is set to be inconsistent with the preset position, it is also necessary to obtain a first distance of the target object relative to the preset position, and a second distance between the preset position and the image acquisition unit. Then, based on the pitch angle and the vertical resolution and longitudinal field of view of the image acquisition unit acquired in advance, the longitudinal coordinate of the face of the target object in the viewer's image can be obtained. Through this solution, even if the position of the image acquisition unit is set to be inconsistent with the preset position, the precise position coordinates of the target face can be obtained, and the image acquisition unit and the preset position can also be set according to actual needs, thereby improving the applicability of the present solution.

[0008] In combination with the technical solution provided in the first aspect above, in some possible implementations, acquiring facial image data of the target object from the viewer image based on the horizontal coordinate and the vertical coordinate of the face of the target object in the viewer image includes: acquiring the facial position coordinates of all facial faces in the viewer image to obtain a facial position coordinate set; determining that the face corresponding to the facial position coordinates closest to the target facial position coordinates in the facial position coordinate set is the face of the target object, and the target facial position coordinates include the horizontal coordinate and the vertical coordinate of the face of the target object in the viewer image; and collecting the facial image data of the target object.

[0009] In an embodiment of the present application, by finding the face position coordinates closest to the target face position coordinates from a face position coordinate set that includes the face position coordinates of all faces in the viewer image, and determining the face corresponding to the face position coordinates as the face of the target object, the face image data of the target object can be quickly acquired without having to re-determine the face based on the acquired target face position coordinates, thereby improving the efficiency of acquiring face image data.

[0010] In combination with the technical solution provided in the first aspect above, in some possible implementations, acquiring facial image data of the target object from the viewer image based on the horizontal coordinate and the vertical coordinate of the target object's face in the viewer image includes: collecting facial image data within a preset range of the target face position coordinates, the target face position coordinates including the horizontal coordinate and the vertical coordinate of the target object's face in the viewer image.

[0011] In the embodiment of the present application, by collecting facial image data within a preset range of the target face position coordinates, the facial image data of the target object can be accurately acquired, thereby preventing the acquisition of erroneous facial image data and thereby improving the accuracy of the acquired facial image data.

[0012] In combination with the technical solution provided in the first aspect above, in some possible implementations, determining the compensation weights of the left-eye image data and the right-eye image data with parallax for synthesizing stereoscopic image data based on the compensation value includes: for each sub-pixel point in the specified image data, based on the compensation value and the coordinates of the pixel point to which the sub-pixel point belongs, and the preset weight of the sub-pixel point, calculating the interpolation coefficient of the sub-pixel point, the pixel points include red sub-pixels, green sub-pixels, and blue sub-pixels, the preset weights of the red sub-pixels, the green sub-pixels, and the blue sub-pixels are different, and the specified image data is the left-eye image data or the right-eye image data; based on the interpolation coefficient, obtaining the compensation weight of the sub-pixel point.

[0013] In an embodiment of the present application, for each sub-pixel point in the specified image data, the interpolation coefficient of the sub-pixel point is calculated, and then the compensation weight of the sub-pixel point in the specified image data is obtained based on the interpolation coefficient of the sub-pixel point. By obtaining the compensation weight of each sub-pixel point in the specified image data, the synthesized stereoscopic image data to be displayed can be more accurately focused on the target object, providing the best stereoscopic image data display effect for the target object.

[0014] In combination with the technical solution provided in the first aspect above, in some possible implementations, for each sub-pixel point in the specified image data, the interpolation coefficient of the sub-pixel point is calculated based on the compensation value, the coordinates of the pixel point to which the sub-pixel point belongs, and the preset weight of the sub-pixel point, including: calculating the floating-point number of the sub-pixel point based on the compensation value, the coordinates of the pixel point to which the sub-pixel point belongs, and the preset weight of the sub-pixel point; and obtaining the interpolation coefficient of the sub-pixel point based on the floating-point number of the sub-pixel point.

[0015] In an embodiment of the present application, the floating-point number of the sub-pixel is calculated by using the compensation value, the coordinates of the pixel to which the sub-pixel belongs, and the preset weight of the sub-pixel, and then the interpolation coefficient of the sub-pixel is obtained based on the floating-point number of the sub-pixel. The interpolation coefficient of the sub-pixel can be accurately obtained, thereby providing an accurate interpolation coefficient for the subsequent sub-pixel in the specified image data, thereby improving the accuracy of compensation.

[0016] In a second aspect, the present application provides a stereoscopic image data synthesis device, including an acquisition module and a processing module, the acquisition module being used to acquire an azimuth angle and a pitch angle of a target object relative to a preset position; the processing module being used to obtain a horizontal coordinate of a face of the target object in the viewer image based on the azimuth angle and a horizontal resolution and a horizontal field of view angle of the image acquisition unit acquired in advance; the processing module being used to obtain a vertical coordinate of the face of the target object in the viewer image based on the pitch angle and a vertical resolution and a vertical field of view angle of the image acquisition unit acquired in advance; the acquisition module being used to obtain a vertical coordinate of the face of the target object in the viewer image based on the horizontal coordinate and the vertical coordinate of the face of the target object in the viewer image. The facial image data of the target object is obtained from the viewer image; the processing module is also used to obtain the binocular center point coordinates of the center points of the two eyes of the target object in the viewer image based on the facial image data; the processing module is also used to determine the compensation value of the binocular center point coordinates relative to the preset initial center coordinates; the processing module is also used to determine the respective compensation weights of the left eye image data and the right eye image data with parallax for synthesizing the stereoscopic image data based on the compensation value; the processing module is also used to synthesize based on the left eye graphic data, the right eye image data and the respective compensation weights of the left eye image data and the right eye image data to obtain the stereoscopic image data to be displayed.

[0017] In combination with the technical solution provided in the first aspect above, in some possible implementations, if the position of the image acquisition unit is set not to coincide with the preset position, the acquisition module is specifically used to acquire a first distance of the target object relative to the preset position, and a second distance between the preset position and the image acquisition unit; the processing module is specifically used to obtain the vertical coordinate of the face of the target object in the viewer image based on the pitch angle, the first distance, the second distance, and the pre-acquired vertical resolution and longitudinal field of view of the image acquisition unit.

[0018] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a memory, a processor and an image acquisition unit, wherein the memory is connected to the processor, and the image acquisition unit is connected to the processor; the memory is used to store programs; the image acquisition unit is used to acquire an image of a viewer viewing a stereoscopic composite image; the processor is used to call the program and the viewer image stored in the memory to execute the method provided in the above-mentioned first aspect embodiment and / or any possible implementation method in combination with the above-mentioned first aspect embodiment.

[0019] In a fourth aspect, an embodiment of the present application provides a storage medium having a computer program stored thereon. When the computer program is executed by a computer, the method provided in the embodiment of the first aspect above and / or any possible implementation method in combination with the embodiment of the first aspect above is executed. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 A schematic diagram of a flow chart of a method for synthesizing stereoscopic image data according to an embodiment of the present application;

[0022] Figure 2 A schematic diagram of an azimuth angle and a pitch angle shown in an embodiment of the present application;

[0023] Figure 3 A schematic diagram of a process of obtaining a face of a target object in a viewer image according to an embodiment of the present application;

[0024] Figure 4 A schematic diagram of another process of obtaining a face of a target object in a viewer image according to an embodiment of the present application;

[0025] Figure 5 A structural block diagram of a stereoscopic image data synthesis device shown in an embodiment of the present application;

[0026] Figure 6 A structural block diagram of an electronic device shown in an embodiment of the present application. DETAILED DESCRIPTION

[0027] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0028] It should be noted that similar numbers and letters represent similar items in the following drawings, so once an item is defined in one drawing, it is not necessary to further define and explain it in the subsequent drawings. At the same time, in the description of this application, relational terms such as "first", "second", etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the term "include", "comprise" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements includes not only those elements, but also includes other elements that are not clearly listed, or also includes elements inherent to such process, method, article or equipment. In the absence of more restrictions, the elements limited by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or equipment including the elements.

[0029] Furthermore, the term "and / or" in this application is merely a term used to describe the association relationship between associated objects, indicating that three relationships may exist. For example, A and / or B may represent three situations: A exists alone, A and B exist at the same time, and B exists alone.

[0030] The technical solution of the present application will be described clearly and completely below in conjunction with the accompanying drawings.

[0031] See also Figure 1 , Figure 1 A stereoscopic image data synthesis method is provided in an embodiment of the present application. The stereoscopic image data synthesis method is applied to an electronic device including an image acquisition unit. The image acquisition unit may be a camera or a still camera.

[0032] The following will be combined Figure 1 Describe the steps it involves.

[0033] S100: Acquire facial image data of a target object in a viewer image.

[0034] The viewer image may be captured in real time by the image acquisition unit, or may be captured in advance by the image acquisition unit and stored in a database or disk, and may be directly acquired when needed. The viewer image may include the faces of other viewers in addition to the face of the target object.

[0035] In one implementation, the user may directly specify a person in the viewer image as the target object. In addition, the target object may not be specified by the user. The viewer in the current viewer image may be detected to determine whether the viewer is speaking and the viewer who is speaking is determined as the target object. Alternatively, the body language of each viewer in the viewer image may be analyzed and the viewer who makes a preset body language may be determined as the target object, for example, a viewer who raises his hand may be determined as the target object.

[0036] In one implementation, a specific process for acquiring facial image data of a target object in a viewer image may be, first, acquiring an azimuth and a pitch angle of the target object relative to a preset position; then, based on the azimuth and a horizontal resolution and a horizontal field of view of an image acquisition unit acquired in advance, acquiring a horizontal coordinate of the target object's face in the viewer image; then, based on the pitch angle and a vertical resolution and a vertical field of view of the image acquisition unit acquired in advance, acquiring a vertical coordinate of the target object's face in the viewer image; and acquiring facial image data of the target object from the viewer image according to the horizontal and vertical coordinates of the target object's face in the viewer image.

[0037] Among them, the horizontal resolution of the above-mentioned image acquisition unit represents the number of pixels of the image acquired by the image acquisition unit in the horizontal direction, the vertical resolution represents the number of pixels of the image acquired by the image acquisition unit in the vertical direction, the longitudinal field of view angle is the angle formed by the upper boundary and the lower boundary of the field of view of the image that can be acquired by the image acquisition unit and the position of the image acquisition unit, and similarly, the transverse field of view angle is the angle formed by the left boundary and the right boundary of the field of view of the image that can be acquired by the image acquisition unit and the position of the image acquisition unit. In order to facilitate understanding of the longitudinal field of view angle and the transverse field of view angle of the above-mentioned image acquisition unit, please refer to Figure 2 , Figure 2 Point A in the figure is the location of the image acquisition unit, α is the horizontal field of view, and β is the vertical field of view. The horizontal and vertical field of view of the image acquisition unit can be set according to the seating layout of the viewers, for example, α = 60° and β = 45°. There is no restriction on their specific values ​​here. However, if the horizontal and vertical field of view of the image acquisition unit are set too large, the collected image will be severely distorted, and the line resolution of a single person in the distance will be insufficient, thereby affecting the positioning accuracy.

[0038] To understand the azimuth and elevation angles of the target object relative to the preset position, please continue to refer to Figure 2 , Figure 2 Point O in is the preset position, and θ is the azimuth, which is the angle between the target object and the XOY plane. is the pitch angle, that is, the angle between the target object and the XOZ plane, wherein the XYZ-O coordinate system takes the central axis of the field of view of the image acquisition unit as the Z axis, the X axis is parallel to the field of view of the image acquisition unit, and the X axis is set horizontally, and the Y axis is vertically upward.

[0039] Optionally, when the position of the image acquisition unit is set to coincide with the preset position, HR represents the horizontal resolution of the image acquisition unit, VR represents the vertical resolution of the image acquisition unit, α represents the horizontal field of view angle, β represents the vertical field of view angle, and θ represents the azimuth angle. represents the pitch angle, then the horizontal coordinate u and vertical coordinate v of the target object's face in the viewer's image are:

[0040]

[0041] Optionally, when the position of the image acquisition unit is set to not coincide with the preset position, the horizontal coordinate of the target face is the same as the horizontal coordinate when the position of the image acquisition unit is set to coincide with the preset position. At this time, it is also necessary to obtain a first distance of the target object relative to the preset position and a second distance between the preset position and the image acquisition unit, and then obtain the vertical coordinate of the target object's face in the viewer image based on the pitch angle, the first distance, the second distance and the vertical resolution and longitudinal field of view of the image acquisition unit acquired in advance.

[0042] Wherein, d represents the first distance of the target object relative to the preset position, and x represents the second distance between the preset position and the image acquisition unit. Then, the horizontal coordinate u and the vertical coordinate v of the face of the target object in the viewer's image are:

[0043]

[0044] Optionally, the azimuth and pitch angles of the target object relative to a preset position, as well as the first distance of the target object relative to the preset position, can be obtained by using a positioning device such as a sound source positioning unit, an infrared positioning unit, etc. that can be used to obtain the position of the target object. In this case, the position of the positioning device is the preset position.

[0045] Among them, when the sound source localization unit is a microphone array and a corresponding signal processing circuit, when the microphone array is used to obtain the azimuth and pitch angle of the target object relative to the preset position, and the first distance of the target object relative to the preset position, the image acquisition unit can be set at the geometric center of the microphone array, and the geometric center of the microphone array is the preset position. At this time, the position of the image acquisition unit coincides with the preset position, and only the azimuth and pitch angle of the target object relative to the microphone array need to be obtained to calculate the horizontal coordinate and vertical coordinate of the target face position; when the image acquisition unit is not set at the geometric center of the microphone array, it is also necessary to obtain the distance between the microphone array and the image acquisition unit, and obtain the distance between the target object and the microphone array through the microphone array, and then calculate the horizontal coordinate and vertical coordinate of the target face position. Among them, the distance between the sound source localization unit and the image acquisition unit is the distance between the geometric center point of the microphone array and the image acquisition unit. At this time, the person who speaks among the viewers is the target object.

[0046] In one implementation, after obtaining the horizontal and vertical coordinates of the target object's face in the viewer image, it is also necessary to obtain the face position coordinates of all faces in the viewer image to obtain a face position coordinate set; then, the face corresponding to the face position coordinates closest to the target face position coordinates in the face position coordinate set is determined to be the face of the target object, and the target face position coordinates include the horizontal and vertical coordinates of the target object's face in the viewer image; and then the face image data of the target object is collected.

[0047] When a microphone array is used as a positioning unit, the specific process of determining the target face using the above method is as follows: Figure 3 As shown, the microphone array is first initialized, and then the microphone array is used to start monitoring, and effective sound is confirmed from the monitored sound. The object of the effective sound is taken as the target object, and the position information of the target object is obtained. The position information includes azimuth, pitch angle, and distance, and then the face coordinates of the target object face in the viewer image are calculated. At the same time, the camera is initialized, and the frame image captured by the camera, that is, the viewer image, is obtained. The face is detected from the frame image, and the face coordinates of each face are obtained to obtain a face coordinate set, and then the face of the target object is filtered out from the face coordinate set according to the face coordinates of the target object face in the viewer image.

[0048] The facial position coordinates of all faces in the viewer image may be acquired by using facial recognition technology. The facial recognition technology is well known to those skilled in the art and will not be described in detail here for the sake of brief description.

[0049] In one implementation, after obtaining the horizontal and vertical coordinates of the target object's face in the viewer image, facial image data within a preset range of the target face position coordinates is collected, and the target face position coordinates include the horizontal and vertical coordinates of the target object's face in the viewer image.

[0050] The preset range may be a rectangular area with a width of w and a height of h, and the specific values ​​of the width w and the height h are based on being able to effectively distinguish two people sitting adjacent to each other, that is, to ensure that two or more faces will not be retrieved within a preset range.

[0051] When a microphone array is used as a positioning unit, the specific process of determining the target face using the above method is as follows: Figure 4 As shown, the microphone array is first initialized, and then the microphone array is used to start monitoring, and effective sound is confirmed from the monitored sound. The object of the effective sound is taken as the target object, and the position information of the target object is obtained. The position information includes azimuth, pitch angle, and distance, and then the face coordinates of the target object face in the viewer image are calculated. At the same time, the camera is initialized, and the frame image captured by the camera is obtained, that is, the viewer image, and the face coordinates of the target object face in the viewer image are used to crop the image, and the face in the cropped image is detected, and the face is the face of the target object.

[0052] Optionally, when the target object is positioned using a sound source localization unit and the target object is close to an electronic device to which the stereoscopic image synthesis method is applied, considering that there is still an upper and lower position deviation between the vocal organ and the human eye at this time, the area of ​​the preset range should be symmetrical around the target face position coordinates, and the area above the target face position coordinates should be slightly larger than the area below.

[0053] S200: Obtaining the coordinates of the center points of the eyes of the target object in the viewer image according to the face image data.

[0054] After obtaining the face image data of the target object in the viewer image, the coordinates of the center points of the two eyes of the target object in the viewer image are obtained based on the face image data.

[0055] Among them, a cascade classifier pattern recognition algorithm can be used to perform face detection on the acquired face image data according to the face template and the gradient change window, and after the face is acquired, the eye location is performed to obtain the coordinates of both eyes, and then the coordinates of the center points of both eyes are calculated based on the coordinates of both eyes. Alternatively, a deep learning method can be used to detect the face image data to obtain the coordinates of the center points of both eyes. The specific method and principle of using a cascade classifier pattern recognition algorithm and a deep learning method to detect the face image data and obtain the coordinates of the center points of both eyes are well known to those skilled in the art, and for the sake of brief description, they will not be repeated here.

[0056] S300: Determine compensation values ​​of binocular center point coordinates relative to preset initial center coordinates.

[0057] After obtaining the binocular center coordinates of the target object, the compensation values ​​of the binocular center coordinates relative to the preset initial center coordinates are determined, wherein the preset initial center point is the focus position of the stereoscopic image to be displayed obtained by directly synthesizing the left eye image data and the right eye image data, that is, the position with the best viewing effect of the stereoscopic image to be displayed.

[0058] Wherein, (u1, v1) represents the binocular center coordinates of the target object, and (u0, v0) represents the preset initial center coordinates, then the compensation value r=[u-u0+(v-v0)*c3] / pd, wherein c3 is the number of sub-pixels covered by the grating of the naked-eye 3D display in each period in the horizontal direction, and pd is the binocular distance of the target object in pixels in the viewer image captured by the image acquisition unit.

[0059] S400: Determine compensation weights of left-eye image data and right-eye image data with parallax for synthesizing stereoscopic image data according to the compensation value.

[0060] After determining the compensation value of the coordinates of the center points of both eyes relative to the preset initial center coordinates, the compensation weights of the left eye image data and the right eye image data with parallax used to synthesize the stereo image data are determined according to the compensation value. The left eye image data and the right eye image data can be video streams obtained by real-time shooting by a stereo camera, or can be pre-acquired and stored in a database or disk, and can be directly acquired when needed, or can also be acquired in real time from an external network.

[0061] In one embodiment, the process of determining the compensation weights of left-eye image data and right-eye image data with parallax for synthesizing stereoscopic image data based on the compensation value can be: first, for each sub-pixel point in the specified image data, based on the compensation value and the coordinates of the pixel point to which the sub-pixel point belongs, and the preset weight of the sub-pixel point, the interpolation coefficient of the sub-pixel point is calculated, and then based on the interpolation coefficient of the sub-pixel point, the compensation weight of the sub-pixel point is obtained, wherein the pixel points include red sub-pixels, green sub-pixels, and blue sub-pixels, and the preset weights of the red sub-pixels, the green sub-pixels, and the blue sub-pixels are different, and the specified image data is left-eye image data or right-eye image data.

[0062] The specific process of obtaining the compensation weight of the sub-pixel point based on the interpolation coefficient is as follows: if the designated image data is left-eye image data, the compensation weight of the sub-pixel point is the interpolation coefficient of the sub-pixel point; if the designated image data is right-eye image data, the compensation weight of the sub-pixel point is the difference between 1 and the interpolation coefficient of the sub-pixel point. For ease of understanding, s is used to represent the interpolation coefficient, then the compensation weight of the sub-pixel point in the left-eye image data is s, and the compensation weight of the sub-pixel point in the right-eye image data is 1-s.

[0063] In one implementation, for each sub-pixel in the specified image data, based on the compensation value and the coordinates of the pixel to which the sub-pixel belongs, and the preset weight of the sub-pixel, the specific process of calculating the interpolation coefficient of the sub-pixel may be, first, based on the compensation value and the coordinates of the pixel to which the sub-pixel belongs, and the preset weight of the sub-pixel, calculating the floating point number of the sub-pixel, and then based on the floating point number of the sub-pixel, obtaining the interpolation coefficient of the sub-pixel. Among them, the preset weights of the same type of sub-pixels of different pixels are the same, that is, the type of the sub-pixel can be determined by the preset weights, for example, a preset weight of 0 represents a red sub-pixel, a preset weight of 1 represents a blue sub-pixel, and a preset weight of 2 represents a green sub-pixel.

[0064] For example, r represents the compensation value, (i, j) represents the coordinates of the pixel to which the sub-pixel belongs, k represents the preset weight of the sub-pixel, k is 0 represents the red sub-pixel, k is 1 represents the blue sub-pixel, and k is 2 represents the green sub-pixel. Then the floating point number t = (3*i+k-3*j*c1+c2) / c3+r, where c1 represents the tangent value of the angle between the grating period arrangement direction of the naked eye 3D display and the horizontal side of the display screen, c2 represents the phase difference between the grating unit of the naked eye 3D display and a certain reference pixel of the display screen, and c3 is the number of sub-pixels covered by the grating of the naked eye 3D display in each period in the horizontal direction. Correspondingly, the interpolation coefficient s = t-floor(t), where floor(t) represents the rounding down of the floating point number t.

[0065] S500: synthesizing the left-eye graphic data, the right-eye image data and the respective compensation weights of the left-eye image data and the right-eye image data to obtain stereoscopic image data to be displayed.

[0066] After obtaining the compensation weights of the left-eye image data and the right-eye image data, the left-eye graphic data, the right-eye image data and the compensation weights of the left-eye image data and the right-eye image data are synthesized to obtain the stereoscopic image data to be displayed.

[0067] For example, if s is used to represent the interpolation coefficient, the compensation weight of the sub-pixel point in the left-eye image data is s, and the compensation weight of the sub-pixel point in the right-eye image data is 1-s. L (i, j, k) represents the color grayscale of a sub-pixel in the left eye image data, and G R (i, j, k) represents the color grayscale of the corresponding sub-pixel in the right image data, then the color grayscale of the synthesized sub-pixel is G(i, j, k) = G L (i,j,k)*s+G R (i,j,k)*(1-s).

[0068] In one implementation, after obtaining the stereoscopic image data to be displayed, the stereoscopic image data to be displayed is sent to a third-party electronic device for display, wherein the third-party electronic device includes a 3D display module, the 3D display module is used to display the stereoscopic image data to be displayed, and the position of the third-party electronic device and the electronic device applying the stereoscopic image data synthesis method is relatively fixed.

[0069] In one embodiment, an electronic device using the stereoscopic image data synthesis method includes a 3D display module for displaying the stereoscopic image data to be displayed. At this time, after the stereoscopic image data to be displayed is obtained, the stereoscopic image data to be displayed is displayed through the 3D display module.

[0070] See also Figure 5 , Figure 5 A stereoscopic image data synthesis device 100 provided in an embodiment of the present application includes an acquisition module 110 and a processing module 120 .

[0071] The acquisition module 110 is used to acquire the azimuth angle and the pitch angle of the target object relative to a preset position.

[0072] The processing module 120 is used to obtain the horizontal coordinate of the face of the target object in the viewer image based on the azimuth angle and the horizontal resolution transverse field angle of the image acquisition unit obtained in advance.

[0073] The processing module 120 is further configured to obtain the vertical coordinate of the face of the target object in the viewer image based on the pitch angle and the pre-acquired vertical resolution and vertical field of view of the image acquisition unit.

[0074] The acquisition module 110 is further configured to acquire the face image data of the target object from the viewer image according to the horizontal coordinate and the vertical coordinate of the face of the target object in the viewer image.

[0075] The processing module 120 is further configured to obtain the coordinates of the center points of the eyes of the target object in the viewer image according to the face image data.

[0076] The processing module 120 is further configured to determine a compensation value of the binocular center point coordinates relative to a preset initial center coordinate.

[0077] The processing module 120 is further configured to determine compensation weights of left-eye image data and right-eye image data with parallax for synthesizing stereoscopic image data according to the compensation value.

[0078] The processing module 120 is further configured to synthesize the left-eye graphic data, the right-eye image data, and respective compensation weights of the left-eye image data and the right-eye image data to obtain stereoscopic image data to be displayed.

[0079] If the position of the image acquisition unit is set not to coincide with the preset position, the acquisition module 110 is specifically used to acquire a first distance of the target object relative to the preset position, and a second distance between the preset position and the image acquisition unit; the processing module 120 is specifically used to obtain the vertical coordinate of the face of the target object in the viewer image based on the pitch angle, the first distance, the second distance and the vertical resolution and longitudinal field of view of the image acquisition unit acquired in advance.

[0080] The processing module 120 is specifically used to obtain the face position coordinates of all faces in the viewer image to obtain a face position coordinate set; determine that the face corresponding to the face position coordinates closest to the target face position coordinates in the face position coordinate set is the face of the target object, and the target face position coordinates include the horizontal coordinates and the vertical coordinates of the face of the target object in the viewer image; and collect face image data of the target object.

[0081] The processing module 120 is specifically configured to collect face image data within a preset range of the target face position coordinates, wherein the target face position coordinates include the horizontal coordinate and the vertical coordinate of the face of the target object in the viewer image.

[0082] The processing module 120 is specifically used to calculate the interpolation coefficient of each sub-pixel point in the specified image data based on the compensation value and the coordinates of the pixel point to which the sub-pixel point belongs, and the preset weight of the sub-pixel point, the pixel point includes a red sub-pixel point, a green sub-pixel point, and a blue sub-pixel point, and the preset weights of the red sub-pixel point, the green sub-pixel point, and the blue sub-pixel point are different, and the specified image data is the left eye image data or the right eye image data; based on the interpolation coefficient, the compensation weight of the sub-pixel point is obtained.

[0083] The processing module 120 is specifically used to calculate the floating point number of the sub-pixel point based on the compensation value, the coordinates of the pixel point to which the sub-pixel point belongs, and the preset weight of the sub-pixel point; and obtain the interpolation coefficient of the sub-pixel point based on the floating point number of the sub-pixel point.

[0084] The implementation principle and technical effects of the stereoscopic image data synthesis device 100 provided in the embodiment of the present application are the same as those of the aforementioned stereoscopic image data synthesis method embodiment. For the sake of brief description, for matters not mentioned in the device embodiment, reference may be made to the corresponding contents in the aforementioned stereoscopic image data synthesis method embodiment.

[0085] See also Figure 6 , which is an electronic device 200 provided in an embodiment of the present application. The electronic device 200 includes: a transceiver 210, a memory 220, a communication bus 230, a processor 240, and an image acquisition unit (not shown in the figure).

[0086] In one embodiment, the electronic device 200 also includes a positioning unit, which is arranged at a preset position. When the position of the image acquisition unit coincides with the preset position, the positioning unit is used to obtain the azimuth and pitch angle of the target object relative to the preset position; when the position of the image acquisition unit does not coincide with the preset position, the positioning unit is used to obtain the azimuth and pitch angle of the target object relative to the preset position, a first distance of the target object relative to the preset position, and a second distance between the preset position and the image acquisition unit.

[0087] In one implementation manner, the electronic device 200 further includes a 3D display module, and the 3D display module is used to display the stereoscopic image data to be displayed after obtaining the stereoscopic image data to be displayed.

[0088] The transceiver 210, the memory 220, the processor 240, and the image acquisition unit are directly or indirectly electrically connected to each other to achieve data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses 230 or signal lines. Among them, the image acquisition unit is used to capture the viewer image of the viewer watching the stereoscopic composite image and send the viewer image to the processor 240, and the transceiver 210 is used to send and receive data. The memory 220 is used to store computer programs, such as storing Figure 5 The software function modules shown in the figure are the stereo image data synthesis device 100. The stereo image data synthesis device 100 includes at least one software function module that can be stored in the memory 220 in the form of software or firmware or fixed in the operating system (OS) of the electronic device 200. The processor 240 is used to execute the executable modules stored in the memory 220, such as the software function modules or computer programs included in the stereo image data synthesis device 100. At this time, the processor 240 is used to obtain the azimuth and pitch angle of the target object relative to the preset position; obtain the horizontal coordinate of the face of the target object in the viewer image based on the azimuth and the horizontal resolution and horizontal field of view of the image acquisition unit obtained in advance; obtain the vertical coordinate of the face of the target object in the viewer image based on the pitch angle and the vertical resolution and vertical field of view of the image acquisition unit obtained in advance; obtain the facial image data of the target object from the viewer image according to the horizontal coordinate and the vertical coordinate of the face of the target object in the viewer image; obtain the binocular center point coordinates of the center points of the two eyes of the target object in the viewer image according to the facial image data; determine the compensation value of the binocular center point coordinates relative to the preset initial center coordinates; determine the compensation weights of the left eye image data and the right eye image data with parallax for synthesizing the stereoscopic image data according to the compensation value; synthesize according to the left eye graphic data, the right eye image data and the compensation weights of the left eye image data and the right eye image data to obtain the stereoscopic image data to be displayed.

[0089] Among them, the memory 220 can be, but is not limited to, random access memory (Random Access Memory, RAM), read only memory (Read Only Memory, ROM), programmable read-only memory (Programmable Read-Only Memory, PROM), erasable programmable read-only memory (Erasable Programmable Read-Only Memory, EPROM), electrically erasable read-only memory (Electric Erasable Programmable Read-Only Memory, EEPROM), etc.

[0090] The processor 240 may be an integrated circuit chip with signal processing capabilities. The above-mentioned processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The various methods, steps and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or the processor 240 may also be any conventional processor, etc.

[0091] The electronic device 200 mentioned above includes but is not limited to a personal computer, a server, and an electronic device including a 3D display module.

[0092] The embodiment of the present application also provides a non-volatile computer-readable storage medium (hereinafter referred to as storage medium), on which a computer program is stored, and when the computer program is run by a computer such as the above-mentioned electronic device 200, the above-mentioned stereoscopic image data synthesis method is executed. The computer-readable storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0093] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for synthesizing stereoscopic image data, characterized in that: Applied to an electronic device comprising an image acquisition unit, the image acquisition unit being used to acquire an image of a viewer viewing a stereoscopic composite image, the method comprising: Obtaining the azimuth and elevation angles of the target object relative to a preset position; Based on the azimuth angle and the horizontal resolution transverse field angle of the image acquisition unit acquired in advance, obtaining the horizontal coordinate of the face of the target object in the viewer image; Obtaining the vertical coordinate of the face of the target object in the viewer image based on the pitch angle and the vertical resolution and the vertical field angle of the image acquisition unit acquired in advance; Acquire facial image data of the target object from the viewer image according to the horizontal coordinate and the vertical coordinate of the face of the target object in the viewer image; According to the face image data, obtaining the coordinates of the center points of the eyes of the target object in the viewer image; Determine a compensation value of the binocular center point coordinates relative to a preset initial center coordinate; wherein the preset initial center point is a focus position of a stereoscopic image to be displayed obtained by directly synthesizing left-eye image data and right-eye image data; Determine, according to the compensation value, respective compensation weights of left-eye image data and right-eye image data with parallax for synthesizing stereoscopic image data; Synthesize the left-eye image data, the right-eye image data, and their respective compensation weights to obtain stereoscopic image data to be displayed; The step of determining the compensation weights of the left-eye image data and the right-eye image data with parallax for synthesizing the stereoscopic image data according to the compensation value comprises: For each sub-pixel point in the specified image data, a floating point number of the sub-pixel point is calculated based on the compensation value, the coordinates of the pixel point to which the sub-pixel point belongs, and the preset weight of the sub-pixel point; the pixel points include red sub-pixels, green sub-pixels, and blue sub-pixels, the preset weights of the red sub-pixels, the green sub-pixels, and the blue sub-pixels are different, and the specified image data is the left-eye image data or the right-eye image data; Obtaining an interpolation coefficient of the sub-pixel based on the floating-point number of the sub-pixel; Based on the interpolation coefficient, a compensation weight of the sub-pixel is obtained.

2. The method according to claim 1, characterized in that If the position of the image acquisition unit is set to not coincide with the preset position, the method further includes: Acquire a first distance of the target object relative to the preset position, and a second distance between the preset position and the image acquisition unit; Based on the pitch angle and the pre-acquired vertical resolution and longitudinal field angle of the image acquisition unit, the longitudinal coordinate of the face of the target object in the viewer image is obtained, including: The vertical coordinate of the face of the target object in the viewer image is obtained based on the pitch angle, the first distance, the second distance, and the pre-acquired vertical resolution and longitudinal field angle of the image acquisition unit.

3. The method according to claim 1, characterized in that The step of acquiring facial image data of the target object from the viewer image according to the horizontal coordinate and the vertical coordinate of the face of the target object in the viewer image comprises: Obtaining face position coordinates of all faces in the viewer image to obtain a face position coordinate set; Determine that a face corresponding to face position coordinates closest to the target face position coordinates in the face position coordinate set is the face of the target object, wherein the target face position coordinates include the horizontal coordinate and the vertical coordinate of the face of the target object in the viewer image; Collect facial image data of the target object.

4. The method according to claim 1, characterized in that: The step of acquiring facial image data of the target object from the viewer image according to the horizontal coordinate and the vertical coordinate of the face of the target object in the viewer image comprises: The face image data within a preset range of the target face position coordinates are collected, wherein the target face position coordinates include the horizontal coordinate and the vertical coordinate of the face of the target object in the viewer image.

5. A stereoscopic image data synthesis device, characterized in that: Deployed in an electronic device including an image acquisition unit, the image acquisition unit is used to acquire an image of a viewer viewing a stereoscopic synthetic image, and the stereoscopic image data synthesis device comprises: An acquisition module is used to acquire the azimuth and elevation angle of the target object relative to a preset position; A processing module, configured to obtain a horizontal coordinate of the face of the target object in the viewer image based on the azimuth angle and a horizontal resolution transverse field angle of the image acquisition unit acquired in advance; The processing module is further used to obtain the vertical coordinate of the face of the target object in the viewer image based on the pitch angle and the vertical resolution and the vertical field angle of the image acquisition unit acquired in advance; The acquisition module is further used to acquire the face image data of the target object from the viewer image according to the horizontal coordinate and the vertical coordinate of the face of the target object in the viewer image; The processing module is further used to obtain the coordinates of the center points of the eyes of the target object in the viewer image according to the face image data; The processing module is further used to determine the compensation value of the binocular center point coordinates relative to the preset initial center coordinates; wherein the preset initial center point is the focus position of the stereoscopic image to be displayed obtained by directly synthesizing the left eye image data and the right eye image data; The processing module is further used to determine compensation weights of left-eye image data and right-eye image data with parallax for synthesizing stereoscopic image data according to the compensation value; The processing module is further used to synthesize the left-eye image data, the right-eye image data and the compensation weights of the left-eye image data and the right-eye image data to obtain the stereoscopic image data to be displayed; The processing module is specifically used to calculate, for each sub-pixel point in the specified image data, a floating-point number of the sub-pixel point based on the compensation value, the coordinates of the pixel point to which the sub-pixel point belongs, and a preset weight of the sub-pixel point; the pixel points include red sub-pixels, green sub-pixels, and blue sub-pixels, and the preset weights of the red sub-pixels, the green sub-pixels, and the blue sub-pixels are different, and the specified image data is the left-eye image data or the right-eye image data; based on the floating-point number of the sub-pixel point, an interpolation coefficient of the sub-pixel point is obtained; based on the interpolation coefficient, a compensation weight of the sub-pixel point is obtained.

6. The stereoscopic image data synthesis device according to claim 5, characterized in that: If the position of the image acquisition unit is set to be not coincident with the preset position, the acquisition module is specifically used to acquire a first distance of the target object relative to the preset position, and a second distance between the preset position and the image acquisition unit; The processing module is specifically used to obtain the vertical coordinate of the face of the target object in the viewer image based on the pitch angle, the first distance, the second distance, and the pre-acquired vertical resolution and longitudinal field of view of the image acquisition unit.

7. An electronic device, characterized in that: include: A memory, a processor and an image acquisition unit, wherein the memory is connected to the processor, and the image acquisition unit is connected to the processor; The memory is used to store programs; The image acquisition unit is used to acquire an image of a viewer viewing a stereoscopic composite image, and send the viewer image to the processor; The processor is used to call the program stored in the memory to execute the method according to any one of claims 1 to 4.

8. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a computer, the method according to any one of claims 1 to 4 is executed.

Images