Wide-viewing-angle stereoscopic camera device and depth image processing method using same

Abstract

The invention discloses a wide-viewing-angle stereoscopic camera device and a depth image processing method using the same. A stereoscopic camera device according to one embodiment of the present invention comprises: a receiving unit for receiving a first image and a second image of a subject captured by a first lens and a second lens which are vertically arranged; a conversion unit for converting the received first image and second image by using a map projection method; and a processing unit that extracts the depth of the subject by performing vertical stereo matching on the first image and the second image converted by a map projection method.

Description

[0001] This application is a divisional application of the invention patent application filed on June 28, 2022, with application number 202180007726.6 and entitled "Wide-angle stereo camera device and depth image processing method using the same". Technical Field

[0002] The present invention relates to a wide-viewing angle stereo camera device and a depth image processing method using the same, and more specifically, to a stereo camera device and a depth image processing method using the same, that is, by vertically configuring two lenses (e.g., fisheye lenses) to widen the horizontal and vertical viewing angles, thereby having a wide viewing angle. Background Art

[0003] In order to realize the human-computer interface (Human-Computer Interface), research on RGB-D cameras that simultaneously output color images and depth maps as three-dimensional (3D) information is very active. The RGB-D camera Kinect sensor promoted by Microsoft for its own games provides the function of recognizing the body movements of multiple people. Therefore, it is not only used in games, but also in various human-computer interfaces. After that, various RGB-D camera products have been launched, expanding the scope of application to 3D content production, autonomous driving, robots and other fields.

[0004] According to the method of obtaining depth information, RGB-D cameras can be divided into stereo cameras and time of flight (TOF) cameras. A typical stereo camera consists of two cameras separated by the same baseline distance, and obtains depth information through the inconsistency of the viewpoints between the cameras. In order to obtain depth information, a stereo matching process is required to compare the images of the lenses on both sides, and pattern light or structured light is also used to improve the speed and accuracy of the stereo matching process. A time of flight camera is a camera that uses the time it takes for the light irradiated by the laser to be reflected back from the subject to calculate the distance and provide two-dimensional (2D) images and depth information. Time of flight cameras usually provide more accurate distance information than stereo methods, but the spatial resolution provided is relatively low, and there are also limitations in measuring distance and usage environment.

[0005] General cameras use lenses designed for rectilinear projection. In rectilinear projection, straight lines in the real world are also displayed as straight lines in the image, so images similar to human perception can be obtained, but there is a problem that the wider the camera's viewing angle, the objects located on the periphery of the image appear relatively larger than the objects in the center. Generally, a rectilinear projection lens is used when the horizontal viewing angle does not exceed about 120 degrees, and a fisheye lens is used when a wider viewing angle is required. The fisheye lens projects the real world into the image as if there is barrel distortion, so it can capture a viewing angle of more than 180 degrees in the image.

[0006] As the use environment of RGB-D cameras expands to mobile objects such as robots, drones, and autonomous vehicles, the demand for wide viewing angles is also increasing. However, since most RGB-D cameras use rectilinear projection lenses, it is difficult to provide a horizontal viewing angle of more than 120 degrees. Although some stereo cameras use fisheye lenses to provide a wider viewing angle, there is a disadvantage that fisheye lens images are difficult to perform effective stereo matching through one-dimensional (1D) search like rectilinear projection images. In addition, even if a fisheye lens is used, it is impossible to obtain distance information in the same direction as the baseline of the stereo camera, so the horizontal viewing angle is limited to 180 degrees. Summary of the invention

[0007] Technical issues

[0008] An embodiment of the present invention provides a stereo camera device and a depth image processing method using the same, that is, by vertically configuring two lenses (e.g., fisheye lenses) to widen the horizontal viewing angle to the maximum horizontal viewing angle of the lens and the vertical viewing angle to 180 degrees, thereby having a wide viewing angle.

[0009] Furthermore, an embodiment of the present invention provides a human-machine interface (HMI) system based on a wide-viewing angle stereo camera and a depth image processing method using the same, that is, the described stereo camera device can be used to identify the whole body movements of a user within a distance of approximately 1 meter.

[0010] Furthermore, an embodiment of the present invention provides a first-person vision system based on a wide-viewing angle stereo camera and a depth image processing method using the same, that is, the described stereo camera device can be used to recognize the user's movements or the user's surrounding environment.

[0011] Furthermore, the embodiment of the present invention provides a personnel tracking system and method based on a wide-viewing angle stereo camera, that is, the described stereo camera device can be used to identify and track personnel at a close distance.

[0012] Technical Solution

[0013] A stereo camera device according to one embodiment of the present invention comprises: a receiving unit, receiving a first image and a second image of a subject captured by a first lens and a second lens arranged vertically; a conversion unit, converting the received first image and second image by using a map projection method; and a processing unit, performing vertical stereo matching on the first image and second image converted by using the map projection method to extract the depth of the subject.

[0014] The conversion unit may convert the first image and the second image using an equirectangular projection method.

[0015] The conversion unit may perform rotational transformation using a spherical coordinate system so that an epipolar line is consistent with a meridian of a spherical surface.

[0016] The processing unit may perform stereo matching by searching the first image and the second image converted by a map projection method along a vertical line corresponding to a meridian.

[0017] The first lens and the second lens may include fisheye lenses.

[0018] The first lens and the second lens may be fisheye lenses with a horizontal viewing angle of more than 180 degrees.

[0019] The processing unit may use the extracted depth of the subject to obtain an image of the subject having a horizontal wide viewing angle of more than 180 degrees.

[0020] The above-mentioned receiving unit may include an image sensor for photographing the above-mentioned first image and an image sensor for photographing the above-mentioned second image. When the image sensors for photographing the above-mentioned first image and the second image are in a rectangular shape with a horizontal length greater than a vertical length, they can be horizontally arranged in a manner to obtain a horizontal viewing angle wider than a vertical viewing angle.

[0021] The stereo camera device may further include a visual processing unit, which utilizes the depth map of the subject extracted from the stereo camera device to identify the user's movements and the surrounding environment including the user's surrounding objects and terrain, and provides the user with the identified user's movements and surrounding environment.

[0022] The above-mentioned stereo camera device may also include a control unit, which uses the depth map of the above-mentioned subject extracted from the above-mentioned stereo camera device to identify the person and the distance between the person, and controls the person in a manner of tracking the person at a specified distance using the identified person and the distance between the person.

[0023] The present invention is characterized in that the control unit can use the depth map of the subject to identify the movement direction of the person and the surrounding environment including the surrounding objects and terrain of the person, so as to track the person and perform control by detecting whether a collision is likely to occur through the movement direction of the person and the surrounding environment and providing feedback on the possibility of collision.

[0024] In a depth image processing method of one embodiment of the present invention, the depth image processing method of a stereo camera device includes the following steps: receiving a first image and a second image of a subject captured by a vertically configured first lens and a second lens; converting the received first image and second image using a map projection method; and performing vertical stereo matching on the first image and second image converted using the map projection method to extract the depth of the subject.

[0025] The above-mentioned depth image processing method may also include the following steps: using the above-mentioned depth map of the extracted subject to identify the movements of the above-mentioned user and the surrounding environment including the surrounding objects and terrain of the above-mentioned user, and providing the above-mentioned user with the identified movements of the above-mentioned user and the surrounding environment.

[0026] The above-mentioned depth image processing method may also include the following steps: using the above-mentioned depth map of the extracted subject to identify the above-mentioned person and the distance between the above-mentioned person, and controlling the above-mentioned person by tracking the above-mentioned person at a specified distance using the identified above-mentioned person and the distance between the above-mentioned person.

[0027] Effects of the Invention

[0028] According to an embodiment of the present invention, the horizontal and vertical viewing angles may be widened by vertically configuring two lenses (eg, fisheye lenses), thereby having a wide viewing angle.

[0029] According to an embodiment of the present invention, a wide viewing angle can be achieved by widening the horizontal and vertical viewing angles to achieve an RGB-D (depth) camera that is wider than the human field of view, and since a wide viewing angle image converted by an equirectangular projection method is utilized, it is easy to detect an object.

[0030] In a fisheye lens image, upright subjects appear tilted or bent in the periphery of the image, but in an image converted using the equirectangular projection method, upright subjects appear upright, making it easier to detect the object.

[0031] The present invention provides a wider viewing angle than that of a human, so the RGB-D camera can be installed on a robot, vehicle, drone or other mobile object for use, and because it can detect the whole body of a person within 1 meter, when it is attached to a tracking robot (Human Following Robot), the whole body can be detected even when it is very close to the tracking target, and when the tracking target changes its moving direction to the left or right, it will not deviate from the field of view, so it can be tracked stably. Therefore, the present invention can be used not only for human-machine interface, but also for human-machine interaction and various mobile objects.

[0032] Furthermore, according to an embodiment of the present invention, a wide-viewing angle stereo camera device may be used to recognize the user's movements or the user's surrounding environment.

[0033] Therefore, according to an embodiment of the present invention, even when a wide-angle stereo camera device is attached to body parts such as the head and torso of a user to shoot the front of the user, due to its wide angle of view, not only the movements of the hands and arms of the user but also the postures and movements of the user can be recognized, and the depth map can be extracted by the wide-angle stereo camera device to detect or recognize the surrounding environment photographed by the stereo camera device, such as detecting or recognizing collisions, whether the ground is flat or tilted, and other terrain.

[0034] When the present invention as described above is applied to devices such as virtual reality (VR), head mounted display (HMD), etc., it is possible to provide information such as the recognized movements of the user and the surrounding environment in a virtual reality manner.

[0035] Furthermore, according to an embodiment of the present invention, a wide-viewing angle stereo camera device may be used to identify and track people at a close distance.

[0036] Therefore, according to the embodiments of the present invention, a personnel tracking system such as a mobile robot can identify and track personnel at a close distance (e.g., within 1 meter), so there is no need for professionals to operate the mobile robot, thereby reducing manpower consumption. That is, since the present invention can capture the whole body of a person at a close distance, various algorithms for detecting and tracking the captured person can be applied.

[0037] According to the embodiment of the present invention, not only can people within a close distance of 1 meter be identified, but also their positions can be accurately determined, so that people moving quickly vertically can be tracked, and people moving quickly horizontally can also be tracked.

[0038] According to an embodiment of the present invention, a person and the surrounding environment including objects and terrain around the person can be identified by utilizing a depth map of the subject extracted by a stereo camera device, thereby detecting whether a collision is likely to occur through the surrounding environment. When it is determined that a collision is about to occur, the possibility of a collision is fed back to control the person tracking system, thereby reducing the possibility of damage to the mobile robot.

[0039] The present invention as described above can be applied to various fields including the logistics field, such as personnel tracking logistics robots, postman tracking robots, personal freight robots, electric golf carts, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 A diagram showing the structure of a stereo camera device according to an embodiment of the present invention.

[0041] Figure 2 An exemplary diagram for explaining the layout of the fisheye lens of the stereo camera device of the present invention and the lateral layout and viewing angle of each image sensor for obtaining a horizontal viewing angle wider than a vertical viewing angle.

[0042] Figure 3 FIG. 1 is an exemplary diagram for explaining the epipolar geometry of the stereo camera device of the present invention.

[0043] Figure 4 An example diagram for illustrating a depth extraction method.

[0044] Figure 5 An example diagram for explaining the conversion of fisheye lens images using equirectangular projection method.

[0045] Figure 6 An example diagram showing the depth for angular differences of the same magnitude.

[0046] Figure 7 An exemplary diagram for explaining the viewing angle of the stereo camera device of the present invention.

[0047] Figure 8 FIG. 4 is a flowchart of a depth image processing method of a stereo camera device according to an embodiment of the present invention.

[0048] Fig. 9 A diagram showing the structure of a first-person vision system of a stereo camera device according to an embodiment of the present invention.

[0049] Fig.10 A diagram showing additional steps of a depth image processing method performed when a stereo camera device according to an embodiment of the present invention operates as a first-person vision system.

[0050] Fig.11 A diagram showing the structure of a person tracking system based on a stereo camera device according to an embodiment of the present invention.

[0051] Fig.12 The diagram shows additional steps of a depth image processing method executed when the stereo camera device according to one embodiment of the present invention operates as a person tracking system. DETAILED DESCRIPTION

[0052] The advantages, features and methods of achieving the present invention can be clearly understood with reference to the accompanying drawings and the detailed embodiments described below. However, the present invention is not limited to the embodiments disclosed below, but can be implemented in various implementations. These embodiments are only provided to make the disclosure of the present invention complete and to enable ordinary technicians in the technical field to which the present invention belongs to fully understand the scope of the present invention. The present invention is only defined by the scope of the attached invention claims.

[0053] The terms used in this specification are intended to illustrate the embodiments rather than to limit the present invention. In this specification, unless otherwise expressly stated in the text, singular expressions include plural expressions. The terms "comprises" and / or "comprising" used in the specification make the mentioned structural elements, steps, operations and / or elements not exclude the existence or addition of one or more other structural elements, steps, operations and / or elements.

[0054] Unless otherwise defined, all terms (including technical and scientific terms) used in this specification may be used according to the meanings commonly understood by those skilled in the art in the art to which the present invention belongs. In addition, for terms defined in commonly used dictionaries, unless otherwise clearly defined, the interpretations made shall not be idealized or excessive.

[0055] Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the accompanying drawings. The same reference numerals are used for the same structural elements in the drawings, and repeated description of the same structural elements will be omitted.

[0056] The main purpose of the embodiments of the present invention is to provide a stereo camera device, that is, to widen the horizontal and vertical viewing angles by vertically configuring two lenses (eg, fisheye lenses) so as to have a wide viewing angle.

[0057] In this case, the present invention can use a map projection method (for example, an equirectangular projection method that displays meridians as straight lines) to convert fisheye images respectively taken by two fisheye lenses, and then perform stereo matching on the fisheye images converted by the equirectangular projection method in the longitudinal or vertical direction to extract the depth of the subject photographed by the fisheye image, and use the depth extracted in this way to obtain a wide-angle image.

[0058] In the stereo camera device of the present invention, since the two fisheye lenses or fisheye lens cameras are vertically configured to face the same direction, if a spherical coordinate system is used, the epipolar line is consistent with the longitude (meridian) of the spherical surface. When the fisheye lens image is converted using the equidistant cylindrical projection method, the longitude is represented as a vertical line in the image, so effective stereo matching can be performed by searching along the vertical line.

[0059] The stereo camera device of the present invention can extract or calculate the depth of the subject based on the angular disparity between images obtained by converting images captured by the upper fisheye lens and the lower fisheye lens using the equirectangular projection method.

[0060] Below, refer to Figures 1 to 8 The present invention will be described.

[0061] Figure 1 This figure is a diagram showing the structure of a stereo camera device according to an embodiment of the present invention, and shows a conceptual structure of a stereo camera device or system. Figure 2 FIG. 1 is an exemplary diagram for explaining the layout of the fisheye lens of the stereo camera device of the present invention and the lateral layout and viewing angle of each image sensor for obtaining a horizontal viewing angle wider than a vertical viewing angle. Figure 3 FIG. 1 is an exemplary diagram for explaining the epipolar geometry of the stereo camera device of the present invention. Figure 4 is an example diagram for illustrating a depth extraction method. Figure 5 This is an example diagram for explaining the conversion of fisheye lens images using equirectangular projection. Figure 6 To illustrate an example diagram of depth for angular differences of the same magnitude, Figure 7 An exemplary diagram for explaining the viewing angle of the stereo camera device of the present invention.

[0062] Reference Figures 1 to 7 The stereo camera device 100 according to the embodiment of the present invention includes a receiving unit 110 , a converting unit 120 and a processing unit 130 .

[0063] The receiving unit 110 receives a first image (or a first fisheye image) and a second image (or a second fisheye image) of a subject captured by a vertically configured first lens (e.g., a first fisheye lens or a first fisheye lens camera) and a second lens (e.g., a second fisheye lens or a second fisheye lens camera).

[0064] For example, Figure 2 As shown, the receiving unit 110 receives a first fisheye image and a second fisheye image of a subject captured by a first fisheye lens and a second fisheye lens arranged vertically. The first fisheye lens and the second fisheye lens are arranged vertically in a manner facing the same direction, and may have a widest horizontal viewing angle and a vertical viewing angle widened to a maximum of 180 degrees. For example, the first fisheye lens and the second fisheye lens arranged vertically may have a horizontal viewing angle of more than 200 degrees and a vertical viewing angle of more than 180 degrees. Of course, each fisheye lens is not limited or restricted to a horizontal viewing angle of more than 200 degrees or a vertical viewing angle of more than 180 degrees, and the horizontal viewing angle and the vertical viewing angle may be different depending on the fisheye lens used.

[0065] In addition, when the first image sensor of the first camera for shooting the first image and the second image sensor of the second camera for shooting the second image of the receiving unit 110 are rectangular in shape with a horizontal length longer than a vertical length, each image sensor can be arranged horizontally to obtain a horizontal viewing angle (or horizontal viewing angle) wider than the vertical viewing angle (or vertical viewing angle). However, the present invention is not limited to this, and the image sensors of the receiving unit 110 can be arranged vertically.

[0066] The conversion unit 120 converts the first fisheye image and the second fisheye image received by the receiving unit 110 by using a map projection method.

[0067] In this case, the conversion unit 120 may convert the first fisheye image and the second fisheye image using an equirectangular projection method.

[0068] Among them, the equirectangular projection method is a type of map projection method, which can represent the latitude and longitude as horizontal lines and vertical lines respectively. Figure 3 As shown, the conversion unit can convert the 220-degree fisheye image into an equirectangular image by using an equirectangular projection method. This equirectangular projection method can be used for 360-degree cameras, etc., to capture a wide viewing angle that cannot be captured in existing rectilinear projection images. In addition, if the equirectangular projection method is used to convert a fisheye lens image with a viewing angle of more than 180 degrees, a distortion phenomenon may occur in which the upper and lower parts of the image are greatly stretched to the left and right.

[0069] In addition, the conversion unit 120 may perform image rectification by rotating the first fisheye image and the second fisheye image to match the direction of the meridian before converting the first fisheye image and the second fisheye image using the equirectangular projection method. If image rectification is performed, the comparison between the two images is simplified to a one-dimensional search, thereby speeding up the stereo matching.

[0070] The processing unit 130 performs stereo matching in the longitudinal or vertical direction on the fisheye images (i.e., the first equirectangular projection image and the second equirectangular projection image) converted by the conversion unit 120 using the equirectangular projection method, thereby extracting the depth or depth map of the subject photographed by the first fisheye lens and the second fisheye lens.

[0071] In this case, if Figure 4 As shown, since the two fisheye lenses or fisheye lens cameras of the processing unit 130 are vertically configured to face the same direction, if a spherical coordinate system is used, the epipolar line is consistent with the meridian of the sphere, and if the equidistant cylindrical projection method is used to convert the fisheye lens image, the meridian is represented as a vertical line in the image, so effective stereo matching can be performed by searching along the vertical line.

[0072] Moreover, if Figure 5 As shown, the processing unit 130 can extract or calculate the depth d of the subject based on the angular disparity between the images converted by the equirectangular projection method using the images captured by the upper fisheye lens and the lower fisheye lens, i.e., φ1-φ0. The depth can be extracted or calculated by the following <Mathematical Formula 1>.

[0073] Mathematical formula 1

[0074]

[0075] From this we can see that even if the angle difference is the same, the depth will change with Furthermore, it can be confirmed that the above and below the same direction as the baseline B (for example is -90 degrees or 90 degrees).

[0076] In this case, if Figure 6 As shown, when the direction facing the camera is set to the Z axis and the direction of the baseline is set to the Y axis, the depth may be different for the same size of angular difference.

[0077] In existing stereoscopic environments with limited viewing angles, depth (or distance) and angular difference are approximately calculated as being inversely proportional. However, when using a wide viewing angle, depth and angular difference have a complex relationship as shown in Formula 1, so this needs to be taken into consideration.

[0078] Furthermore, the processing unit 130 may use the extracted depth of the subject to obtain an image of the subject having a wide viewing angle of more than 180 degrees in the horizontal viewing angle and a maximum vertical viewing angle of 180 degrees.

[0079] In this way, the stereo camera device of the embodiment of the present invention can widen the horizontal and vertical viewing angles by vertically configuring two lenses (e.g., fisheye lenses) to have a wide viewing angle. For example, the present invention can realize a wide viewing angle RGB-D camera device with a maximum vertical viewing angle of 180 degrees and a horizontal viewing angle significantly greater than 180 degrees. Among them, if a 280-degree fisheye lens is used, a horizontal viewing angle of 280 degrees and a vertical viewing angle of 180 degrees can be achieved.

[0080] Furthermore, according to an embodiment of the present invention, the stereo camera device of the embodiment of the present invention can have a wide viewing angle by widening the horizontal viewing angle and the vertical viewing angle, thereby realizing an RGB-D camera with a wider viewing angle than that of a human being and making it easy to detect nearby objects. Figure 7 As shown, since the stereo camera device according to the embodiment of the present invention can easily detect nearby objects, the whole body of a person within 1 meter can be detected.

[0081] Furthermore, when the stereo camera device of the embodiment of the present invention is attached to a tracking robot, the whole body can be detected even if it is very close to the tracking target, and when the tracking target changes its moving direction to the left or right, it will not deviate from the field of view, so that stable tracking can be performed. Therefore, the present invention can be used not only for human-machine interfaces, but also for human-machine interactions and various moving objects.

[0082] As described above, the stereo camera device of the present invention can convert images captured by vertically or longitudinally arranging two fisheye lenses or fisheye lens cameras using the equirectangular projection method, thereby effectively performing stereo matching between the upper image and the lower image.

[0083] Conventional stereo cameras using rectilinear projection images do not differ much when two cameras are arranged horizontally or vertically, so it is only necessary to consider the direction of the subject and arrange them in a convenient direction for installation. However, wide-angle stereo cameras using images converted by equirectangular projection have different image distortion forms depending on the camera's installation direction, such as Figure 6As shown, the accuracy of the depth obtained by stereo matching may also vary, so it is necessary to consider the distortion of the image and the layout of the stereo cameras that can obtain a more accurate depth value of the object.

[0084] In a rectilinear projection image, straight lines in the real world are always displayed as straight lines in the image, so the image will not be distorted due to the setting direction of the camera. However, in an image converted using the equirectangular projection method, although vertical lines in the real world are displayed as straight lines, horizontal lines in the real world are displayed as curves, so the distortion form of the image will differ depending on the setting direction of the camera.

[0085] That is, only when the camera is not tilted to the left or right will an upright subject appear with minimal distortion in an image converted using the equirectangular projection method.

[0086] Therefore, it can be assumed that the camera that uses images converted using the equirectangular projection method is mainly used in an upright state facing forward, and the wide-angle stereo camera with two lenses vertically arranged takes advantage of the vertical arrangement in consideration of the following three contents.

[0087] First, in a stereo environment with a viewing angle of more than 180 degrees, there may be mutual occlusion between the two cameras. For example, the other side camera of the stereo camera can also be seen in the image. Therefore, the vertical configuration of the two cameras can make the occlusion direction to be upward or downward. Generally, compared with the direction toward the ceiling or the direction toward the floor, it is more likely that the valuable subject exists in the front or side direction, so it is advantageous to have the occlusion direction to be upward and downward.

[0088] Second, when using the equirectangular projection method, there is a problem that the closer to the top and bottom of the image corresponding to the pole, the more it is displayed more exaggerated than the actual. The two cameras are arranged vertically to place the direction where this distortion occurs upward and downward.

[0089] Third, in existing stereo environments with limited viewing angles, the depth (or distance) and the size of the parallax are approximately calculated as being inversely proportional, but as the viewing angle widens, even with the same angular difference, the depth gradually decreases from the direction the camera is facing to the baseline direction of the stereo camera, making it difficult to obtain accurate depth. If the two cameras are configured vertically, the direction of the baseline will also be upward and downward, so areas with poor accuracy can be placed above and below.

[0090] The stereo camera device of the present invention can reduce the area where depth cannot be obtained as much as possible by placing the direction in which interference between the two cameras or two lenses is generated in the same direction as the direction in which depth cannot be obtained, and can also expand the viewing angle of the depth map in the horizontal direction to more than 180 degrees. Of course, if the stereo camera device of the present invention uses a 280-degree fisheye lens, a 280-degree depth map in the horizontal direction can be obtained.

[0091] In addition, in the stereo camera device of the embodiment of the present invention, when stereo matching is performed by searching the image of the upper camera with the image of the lower camera as a reference among the upper camera and the lower camera, a hole caused by occlusion will be generated below the object. Compared with the case where the hole is generated on the left or right side of the object in the horizontally placed stereo camera device, even if a hole is generated, it will only be below the object in the present invention, which is more advantageous in detecting objects, etc.

[0092] Figure 8 FIG. 1 is a flowchart of a depth image processing method of a stereo camera device according to an embodiment of the present invention, showing Figures 1 to 7 Workflow diagram of the stereo camera.

[0093] Reference Figure 8 The depth image processing method of the stereo camera device of the embodiment of the present invention includes the following steps: step S810, receiving a first image (for example, a first fisheye image) and a second image (for example, a second fisheye image) of a subject captured by a first lens and a second lens arranged vertically; step S820, if the first fisheye image and the second fisheye image are received through step S810, a map projection method is used to convert the received first fisheye image and the second fisheye image.

[0094] In this case, the first fisheye image and the second fisheye image may be transformed by using an equirectangular projection method in step S820. In addition, a rotation transformation may be performed by using a spherical coordinate system in step S820 so that the epipolar line is consistent with the meridian of the spherical surface.

[0095] If the map projection method is used to convert the first fisheye image and the second fisheye image in step S820, then in steps S830 and S840, the depth of the subject is extracted by vertically performing stereo matching on the first fisheye image and the second fisheye image converted by the map projection method, and the extracted depth of the subject is used to obtain a depth image of the subject with a wide viewing angle having a horizontal viewing angle of more than 180 degrees and a vertical viewing angle of a maximum of 180 degrees.

[0096] In this case, stereo matching may be performed by searching the first image and the second image converted using the map projection method along a vertical line corresponding to the meridian in step S830 .

[0097] Although in Figure 8 The description is omitted in the method, but it constitutes Figure 8 The various steps may include Figures 1 to 7 All the contents described are self-evident to those skilled in the art.

[0098] Furthermore, the stereo camera device can recognize the arm and hand movements of the user being photographed, the user's posture, and the surrounding environment (eg, surrounding terrain, surrounding objects, etc.) and provide such information.

[0099] Below, we will refer to Figures 9 and 10 The present invention described above will be described.

[0100] Fig. 9 A diagram showing the structure of a first-person vision system of a stereo camera device according to an embodiment of the present invention.

[0101] Reference Fig. 9 , the stereo camera device 100 may further include a visual processing unit 910 to operate as a first-person vision system.

[0102] In this case, the stereo camera device 100 can be attached to a body part such as the head or torso of the user, or can be attached to a hat, etc., so as to shoot the front of the user. In addition, although the stereo camera device 100 shoots the front of the user, since the stereo camera device has a wide viewing angle, it can also shoot body parts such as the arms and hands of the user, and can also shoot the surrounding environment (for example, objects, buildings, terrain, etc.). In the present invention, all objects shot in this way are referred to as subjects.

[0103] When the stereo camera device 100 works as a first-person vision system, the visual processing unit 910 uses the depth map of the subject obtained by the stereo camera device 100 to identify the user's movements and the surrounding environment including objects and terrain around the user, and provides the user with the information thus identified, i.e., the user's movements and surrounding environment information.

[0104] For example, the first-person vision system (stereo camera device 100) can provide a field of view similar to that of a human through an RGB-D camera device having a horizontal wide viewing angle of more than 180 degrees, so that the upper, lower, left, and right hand movements of the user wearing the first-person vision system (stereo camera device 100) can all enter the field of view. Therefore, even if the stereo camera device 100 shoots the front of the user, the visual processing unit 910 can recognize the movements of the user's hands and arms, as well as the user's movements, postures, etc.

[0105] Since the stereo camera device 100 can extract a depth map, the visual processing unit 910 can detect the possibility of collision with the user by detecting surrounding objects and provide information about this to the user. When the user has an object in his hand, it can also provide information such as whether the user's hand is behind the object or the object is in the hand by detecting the position of the user's hand and the object. It can also provide information such as surrounding terrain and objects to the user in real time by identifying the surrounding environment.

[0106] As described above, the stereo camera device 100 may further include a visual processing unit 910 to detect the movements of the user's hands and arms or to detect objects such as things at a close distance.

[0107] Fig.10 FIG. 1 is a diagram showing additional steps of a depth image processing method performed when a stereo camera device according to an embodiment of the present invention operates as a first-person vision system, that is, as Fig. 9 The stereo camera device in the first person vision system works in the implementation Figure 8 The steps additionally performed after the step (step S840) of the depth image processing method shown.

[0108] Reference Fig.10 , by referring to Figure 8 After the aforementioned steps S830 and S840 extract the depth or depth map of the subject, the extracted depth map of the subject is used to identify the user's movements and the surrounding environment including objects and terrain around the user, and the identified user's movements and surrounding environment are provided to the user (step S1010).

[0109] In addition, the stereo camera device can identify people and the surrounding environment around them, including objects and terrain, by utilizing the depth map of the subject extracted by the stereo camera device, so as to detect whether a collision is likely to occur through the surrounding environment. When it is determined that a collision is about to occur, the possibility of collision can be fed back to control the personnel tracking system.

[0110] Below, refer to Figure 11 to Figure 12The present invention as described above is described.

[0111] Fig.11 A diagram showing the structure of a person tracking system based on a stereo camera device according to an embodiment of the present invention.

[0112] Reference Fig.11 The stereo camera device 100 may further include a control unit 1110 to operate as a personnel tracking system.

[0113] In this case, since the stereo camera device has a wide viewing angle, the stereo camera device 100 can capture the person and the surrounding environment (eg, objects, buildings, terrain, etc.) that the person tracking system needs to track. In the present invention, all objects captured in this way are referred to as subjects.

[0114] When the stereo camera device 100 works as a personnel tracking system, the control unit 1110 can use the depth map of the subject obtained by the stereo camera device 100 to identify the person to be tracked and the distance to the person to be tracked, and use the identified person and the distance to the person to control the personnel tracking system (stereo camera device 100) to track the person at a specified distance.

[0115] In this case, the control unit 1110 can not only use the depth map to identify the person and the distance between the person and the person, but also identify the person's movement direction, the surrounding environment including things around the person and possible collision objects and terrain, and use the information thus identified (i.e., the person's movement direction and surrounding environment) to detect whether the person tracking system (stereo camera device 100) is likely to collide, and feedback the possibility of collision to control the person tracking system (stereo camera device 100) to track the person.

[0116] The personnel tracking system (stereo camera device 100) provides a field of view similar to that of a human through the stereo camera device 100 (for example, a wide-angle RGB-D camera device with a maximum vertical field of view of 180 degrees and a horizontal field of view greater than 180 degrees), so that personnel can be tracked at a close distance (for example, within 1 meter), and because the whole body of a person can be identified at a close distance, accurate positioning can be achieved, thereby not only tracking personnel moving quickly vertically, but also tracking personnel moving quickly laterally.

[0117] Fig.12 FIG. 1 is a diagram showing additional steps of a depth image processing method executed when a stereo camera device according to an embodiment of the present invention works as a person tracking system, that is, as a Fig.11 The stereo camera device in the personnel tracking system works in Figure 8 The steps additionally performed after the step (step S840) of the depth image processing method shown.

[0118] Reference Fig.12 , by referring to Figure 8 After extracting the depth or depth map of the subject in the aforementioned steps S830 and S840, the extracted depth map of the subject is used to identify the person and the distance between the person, and the person is tracked at a specified distance using the identified person and the distance between the person to control (step S1210).

[0119] In this case, in step S1210, the depth map of the subject can be used to identify the person's movement direction and the surrounding environment including objects and terrain around the person, so as to track the person and control the person by detecting whether a collision is possible through the person's movement direction and the surrounding environment and providing feedback on the possibility of collision.

[0120] The above-mentioned device can be implemented as a combination of hardware structural elements, software structural elements and / or hardware structural elements and software structural elements. For example, the device and structural elements described in the embodiment can be implemented using one or more general-purpose computers or special-purpose computers of a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor (digital signal processor), a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, any other device that can execute and respond to instructions, etc. The processing device can execute an operating system (OS) and one or more software applications running on the above-mentioned operating system. In addition, the processing device can also access, store, operate, process and generate data in response to the execution of the software. For ease of understanding, the processing device can be described as using one element, but it can be understood by ordinary technicians in the field that the processing device includes multiple processing elements and / or various types of processing elements. For example, the processing device can include multiple processors or include a processor and a controller. In addition, other processing configurations are also feasible, such as parallel processors.

[0121] Software may include a computer program, code, instructions, or a combination of more than one of them, and may configure a processing device in a manner to operate as required, or command a processing device independently or collectively. Software and / or data may be embodied as any type of machine, component, physical device, virtual device, computer storage medium, or device to interpret instructions or data by or to provide instructions or data to a processing device. Software may be distributed on networked computer systems and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

[0122] The method of the embodiment can be implemented in the form of program instructions that can be executed by various computer devices, and recorded in a computer-readable medium. The above-mentioned computer-readable medium may contain program instructions, data files, data structures, etc., alone or in combination. The program instructions recorded in the above-mentioned medium may be specially designed and configured for the present embodiment, or may be known and available to technicians in the field of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks, and tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices such as ROMs, RAMs, flash memories, etc., specifically for storing and executing program instructions. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc.

[0123] As described above, although the description is made with reference to limited embodiments and drawings, a person skilled in the art may make various modifications and variations based on the above description. For example, the techniques may be performed in a different order than the method described, and / or the structural elements of the system, structure, device, circuit, etc. may be combined or combined in a different manner than the method described, or even if replaced or substituted by other structural elements or equivalent technical solutions, appropriate results may be achieved.

[0124] Therefore, other implementation modes, other examples and contents equivalent to the protection scope of the present invention also belong to the protection scope of the present invention.

Claims

1. A stereo camera device, characterized in that: include: A receiving unit receives a first image and a second image of a subject captured by a first lens and a second lens arranged vertically; a conversion unit, which converts the received first image and second image by using a map projection method; and The processing unit performs vertical stereo matching on the first image and the second image converted by the map projection method to extract the depth of the object. The conversion unit converts the first image and the second image by using an equirectangular projection method. The conversion unit performs rotational transformation using a spherical coordinate system so that the epipolar line coincides with the meridian of the spherical surface.

2. The stereo camera device according to claim 1, characterized in that The processing unit performs stereo matching by searching the first image and the second image converted by a map projection method along vertical lines corresponding to meridians.

3. The stereo camera device according to claim 1, characterized in that The first lens and the second lens include fisheye lenses.

4. The stereo camera device according to claim 3, characterized in that The first lens and the second lens are fisheye lenses with a horizontal viewing angle of more than 180 degrees.

5. The stereo camera device according to claim 1, characterized in that The processing unit obtains an image of the object having a horizontal wide viewing angle of 180 degrees or more by using the extracted depth of the object.

6. The stereo camera device according to claim 1, characterized in that The receiving unit includes an image sensor for photographing the first image and an image sensor for photographing the second image. When the image sensors used to capture the first image and the second image are rectangular in shape with a horizontal length greater than a vertical length, they are respectively arranged horizontally in such a manner as to obtain a horizontal viewing angle wider than a vertical viewing angle.

7. The stereo camera device according to claim 1, characterized in that It also includes a visual processing unit, which uses the depth map of the subject extracted from the stereo camera device to identify the movements of the user and the surrounding environment including the surrounding objects and terrain of the user, and provides the identified movements of the user and the surrounding environment to the user.

8. The stereo camera device according to claim 1, characterized in that It also includes a control unit, which uses the depth map of the subject extracted from the above-mentioned stereo camera device to identify the person and the distance between the person, and controls the person in a manner of tracking the person at a specified distance using the identified person and the distance between the person.

9. The stereo camera device according to claim 8, characterized in that The control unit uses the depth map of the subject to identify the movement direction of the person and the surrounding environment including the surrounding objects and terrain of the person, and controls the person by detecting whether a collision is likely to occur through the movement direction of the person and the surrounding environment and feeding back the possibility of collision.

10. A depth image processing method for a stereo camera device, characterized in that: The steps include: receiving a first image and a second image of a subject captured by a first lens and a second lens arranged vertically; Converting the received first image and second image using a map projection method; and Performing vertical stereo matching on the first image and the second image converted by the map projection method to extract the depth of the object, The conversion includes converting the first image and the second image using an equirectangular projection method. The above conversion includes a rotation transformation using the spherical coordinate system so that the epipolar line coincides with the meridian of the spherical surface.

11. The depth image processing method according to claim 10, characterized in that: The method further includes the following steps: using the extracted depth map of the subject to identify the user's movements and the surrounding environment including the user's surrounding objects and terrain, and providing the identified user's movements and surrounding environment to the user.

12. The depth image processing method according to claim 10, characterized in that: The method further includes the following steps: using the extracted depth map of the subject to identify the person and the distance between the person and the subject, and controlling the person by tracking the person at a specified distance using the identified person and the distance between the person and the subject.

Images