Cat eye intelligent door lock system and video display method

Abstract

The invention belongs to the technical field of intelligent door locks, and discloses a cat eye intelligent door lock system and a video display method.The system comprises a cat eye monitoring module, a server module and a mobile terminal; the cat eye monitoring module is integrated in the door lock module and is used for acquiring cat eye video data and sending the cat eye video data to the server module; the server module is used for sending the received cat eye video data to the mobile terminal; the mobile terminal is used for performing distortion correction on each image frame of the cat eye video data to obtain a panoramic rectangular image corresponding to each image frame; and receiving a split-screen display instruction, cutting and scaling each panoramic rectangular image according to the split-screen display instruction to obtain multiple groups of video stream data with different sizes, and displaying the video stream data. According to the method and the device, different areas outside the door can be divided and displayed in a magnified manner, so that the requirements of different users on different areas outside the door for different attention degrees are met.

Description

Technical Field

[0001] The present application relates to the technical field of smart door locks, and in particular to a cat-eye smart door lock system and a video display method. Background Art

[0002] Early peepholes were usually set in fixed positions on door panels, resulting in people of different heights seeing different ranges of the environment through the peepholes. Therefore, the peepholes on the market are gradually developing into peephole cameras, allowing users to observe the environment outside the door in real time through their mobile phones. Some manufacturers have integrated peephole cameras and door locks together, further realizing the ability to remotely remind users to check the situation of people outside the door through their mobile phones when someone outside the door tries to unlock the door, thereby improving safety.

[0003] However, existing peephole door locks usually synchronize the original peephole video to the user terminal in real time. As the viewing angle of the peephole gradually increases, the high-latitude area of ​​the peephole video is severely compressed, occupying a very small area of ​​the video area, which makes it easy for users to ignore the surrounding environment when viewing the surveillance remotely; especially when users place storage racks, shoe racks and other sundries at the door, their attention to the sides and bottom of the door will increase accordingly, but the peephole video severely compresses these areas, not only showing a small surveillance area, but also causing video distortion due to compression, which cannot meet surveillance needs. Summary of the invention

[0004] The present application provides a cat-eye smart door lock system and a video display method, which can realize the division and enlarged display of different areas outside the door, so as to meet the different needs of different users for different attention levels to different areas outside the door.

[0005] In a first aspect, an embodiment of the present application provides a cat-eye smart door lock system, comprising:

[0006] Cat's eye monitoring module, server module and mobile terminal;

[0007] The cat's eye monitoring module is integrated in the door lock module to obtain cat's eye video data and send it to the server module;

[0008] The server module is used to send the received Maoyan video data to the mobile terminal;

[0009] The mobile terminal is used to perform distortion correction on each image frame of the cat's eye video data to obtain a panoramic rectangular image corresponding to each image frame; receive a split-screen display instruction, cut and scale each panoramic rectangular image according to the split-screen display instruction, obtain and display multiple groups of video stream data of different sizes.

[0010] Furthermore, the cat's eye monitoring module includes:

[0011] An image acquisition unit, an MCU unit and a fill light unit connected in sequence;

[0012] The image acquisition unit is used to collect cat-eye video data and send it to the MCU unit; and determine whether the proportion of received infrared light reaches a preset night threshold; if so, send a mode switching instruction to the MCU unit;

[0013] The MCU unit is used to send the received cat-eye video data to the server module; and, when receiving a mode switching instruction, control the fill light unit to light up.

[0014] Furthermore, the image acquisition unit includes:

[0015] The panoramic cat-eye lens, dual-channel filter and CIS unit are connected in sequence;

[0016] The dual-channel filter is used to transmit visible light with a wavelength of 400nm-650nm and infrared light with a wavelength of 900nm-1000nm to the CIS unit; the CIS unit generates cat-eye video data based on the received light waves.

[0017] Furthermore, the fill light unit is an infrared fill light with a wavelength of 940nm; the transmittance of the dual-channel filter to infrared light with a wavelength of 900nm-1000nm is 45%.

[0018] Furthermore, the mobile terminal includes:

[0019] A correction unit, used for performing distortion correction on each image frame to obtain a panoramic rectangular image corresponding to each image frame;

[0020] A cutting and scaling unit is used to segment the panoramic rectangular image according to a preset cutting ratio in the split-screen display instruction to obtain multiple groups of video stream data of different sizes; to enlarge or reduce each video stream data according to the preset scaling ratio in the split-screen display instruction, and to place it at a preset display position;

[0021] The format conversion unit is used to convert the enlarged or reduced video stream data into RGB video data, and render and display the RGB video data at a preset display position.

[0022] Furthermore, the correction unit is specifically used for:

[0023] The circular image frame is converted into a rectangular image using an equidistant cylindrical projection model, and the pixel values ​​of the image frame are mapped into the rectangular image using a bilinear algorithm or a bicubic interpolation algorithm to obtain a panoramic rectangular image.

[0024] Furthermore, the mobile terminal also includes:

[0025] A VR display unit is used to receive VR display instructions, map cat's eye video data to a hemispherical surface; determine a viewpoint, a perspective projection matrix and a viewing angle transformation matrix within the hemispherical surface; obtain a first distance and a second distance based on a user's swiping of the viewpoint; determine a first transformation angle and a second transformation angle according to the first distance, the second distance and a screen density of the mobile terminal; determine a target display range according to the first transformation angle, the second transformation angle, the perspective projection matrix and the viewing angle transformation matrix, and render and display the cat's eye video data in the target display range.

[0026] Furthermore, the split-screen display instructions include:

[0027] Up and down split screen instructions and left and right split screen instructions;

[0028] The preset cutting ratio of the top-bottom split screen command is 70% of the upper side and 30% of the lower side of the panoramic rectangular image;

[0029] The preset cutting ratio of the left and right split screen instructions is 70% on the left and 50% on the right of the panoramic rectangular image.

[0030] In a second aspect, the present application provides a video display method for a cat-eye smart door lock, which is applied to a mobile terminal, comprising:

[0031] Receive cat's eye video data; perform distortion correction on each image frame of the cat's eye video data to obtain a panoramic rectangular image corresponding to each image frame;

[0032] Receive a split-screen display instruction; cut and scale each panoramic rectangular image according to the split-screen display instruction to obtain and display multiple groups of video stream data of different sizes.

[0033] Furthermore, the above-mentioned distortion correction is performed on each image frame of the cat's eye video data to obtain a panoramic rectangular image corresponding to each image frame, including:

[0034] The circular image frame is converted into a rectangular image by using an equidistant cylindrical projection model; the pixel values ​​of the image frame are mapped into the rectangular image by using a bilinear algorithm or a bicubic interpolation algorithm to obtain a panoramic rectangular image.

[0035] Furthermore, the above-mentioned panorama rectangular images are cut and scaled according to the split-screen display instruction to obtain multiple groups of video stream data of different sizes, including:

[0036] Cutting the upper 70% of the panoramic rectangular image to obtain a first split-screen image;

[0037] Cutting the lower 30% of the panoramic rectangular image to obtain a second split-screen image;

[0038] Combining the first split-screen images into first video stream data, and combining the second split-screen images into second video stream data;

[0039] Enlarging the first video stream data to a first display size, and reducing the second video stream data to a second display size;

[0040] The data formats of the first video stream data and the second video stream data are both converted into RGB formats.

[0041] Furthermore, the above-mentioned panorama rectangular images are cut and scaled according to the split-screen display instruction to obtain multiple groups of video stream data of different sizes, including:

[0042] Cut the left 70% of the panoramic rectangular image to obtain a first split-screen image;

[0043] Cut the right 50% of the panoramic rectangular image to obtain a second split-screen image;

[0044] Combining the first split-screen images into first video stream data, and combining the second split-screen images into second video stream data;

[0045] Enlarging the first video stream data to a first display size, and reducing the second video stream data to a second display size;

[0046] The data formats of the first video stream data and the second video stream data are both converted into RGB formats.

[0047] Furthermore, the method also includes:

[0048] Receive VR display instructions and map the cat's eye video data to a hemispherical surface;

[0049] Determine the viewpoint, perspective projection matrix and viewing angle transformation matrix within the hemisphere;

[0050] Obtain a first distance and a second distance based on the user's swiping of the viewpoint;

[0051] determining a first transformation angle and a second transformation angle according to the first distance, the second distance and the screen density;

[0052] The target display range is determined according to the first transformation angle, the second transformation angle, the perspective projection matrix and the viewing angle transformation matrix, and the cat's eye video data in the target display range is rendered and displayed.

[0053] In summary, compared with the prior art, the technical solution provided in the embodiment of the present application has at least the following beneficial effects:

[0054] A cat's eye smart door lock system provided in an embodiment of the present application, by performing distortion correction on the received cat's eye video data in a mobile terminal, the original compressed and distorted circular image is corrected into a normal panoramic rectangular image, which can not only restore the real situation outside the door, but also further improve the lens field of view of the cat's eye monitoring module, without worrying about unclear display problems; and then cutting and scaling the panoramic rectangular image according to the user's split-screen display instructions, which can realize the division and enlarged display of different areas outside the door, thereby meeting the different needs of different users for different attention to different areas outside the door. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 A structural diagram of a cat-eye smart door lock system provided as an exemplary embodiment of the present application.

[0056] Figure 2 An internal structural diagram of a cat's eye monitoring module provided as an exemplary embodiment of the present application.

[0057] Figure 3 A spectrum diagram of a dual-channel filter provided for an exemplary embodiment of the present application.

[0058] Figure 4 An internal structure diagram of a mobile terminal provided for an exemplary embodiment of the present application.

[0059] Figure 5 A schematic diagram of the principle of panoramic cat-eye lens imaging provided for an exemplary embodiment of the present application.

[0060] Figure 6 A schematic diagram of a spherical model mapped in distortion correction provided by an exemplary embodiment of the present application.

[0061] Figure 7 A top view of a spherical model mapped in distortion correction provided by an exemplary embodiment of the present application.

[0062] Figure 8 A schematic diagram of a rectangular expansion of an equidistant cylindrical projection model provided for an exemplary embodiment of the present application.

[0063] Fig. 9 A hemispherical schematic diagram of VR display mapping provided for an exemplary embodiment of the present application.

[0064] Fig.10 A flowchart of a video display method for a cat-eye smart door lock provided as an exemplary embodiment of the present application.

[0065] Fig.11 A flowchart of steps for implementing VR display is provided for an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0066] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments.

[0067] Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making any creative work shall fall within the scope of protection of this application.

[0068] See also Figure 1 , the embodiment of the present application provides a cat-eye smart door lock system, including:

[0069] Cat's eye monitoring module, server module and mobile terminal.

[0070] The cat's eye monitoring module is integrated in the door lock module to obtain cat's eye video data and send it to the server module.

[0071] The server module is used to send the received Maoyan video data to the mobile terminal.

[0072] The mobile terminal is used to perform distortion correction on each image frame of the cat's eye video data to obtain a panoramic rectangular image corresponding to each image frame; receive a split-screen display instruction, cut and scale each panoramic rectangular image according to the split-screen display instruction, obtain and display multiple groups of video stream data of different sizes.

[0073] Specifically, for the distortion correction of cat's eye video data, a command trigger may be set separately, that is, the image frame is corrected when the correction command is received, otherwise the original circular cat's eye video data is still displayed.

[0074] In the specific implementation process, before transmitting Maoyan video data, the mobile terminal and server module need to be registered:

[0075] Users can enter personal information such as fingerprints, facial recognition data, passwords, etc. through mobile terminal (such as mobile phone) applications. This information will be encrypted and sent to the server module, and a copy will be kept in the local storage for future verification.

[0076] The server module creates a unique identification code for each mobile terminal and configures the device's network parameters, such as access points, communication protocols, etc. After receiving the Maoyan video data, the server module associates it with the user account.

[0077] The above-mentioned embodiment provides a cat-eye smart door lock system, which corrects the received cat-eye video data in the mobile terminal, and corrects the original compressed and distorted circular image into a normal panoramic rectangular image. It can not only restore the real situation outside the door, but also further improve the lens field of view of the cat-eye monitoring module, without worrying about unclear display problems; and then cuts and scales the panoramic rectangular image according to the user's split-screen display instructions, which can realize the division and focus on different areas outside the door, thereby meeting the different needs of different users for different attention to different areas outside the door.

[0078] In some embodiments, see Figure 2 , the cat's eye monitoring module includes:

[0079] The image acquisition unit, MCU unit and fill light unit are connected in sequence.

[0080] The image acquisition unit is used to collect cat-eye video data and send it to the MCU unit; and to determine whether the proportion of received infrared light reaches a preset night threshold; if reached, a mode switching instruction is sent to the MCU unit.

[0081] The MCU unit is used to send the received cat-eye video data to the server module; and, when receiving a mode switching instruction, control the fill light unit to light up.

[0082] Specifically, the cat-eye smart door lock works 24 hours a day, so when monitoring the situation outside the door, it is also necessary to consider the impact of light changes caused by time on the clarity of the video image; especially at night when there are higher security risks, the monitoring image needs to clearly show the situation outside the door to improve safety at night.

[0083] Therefore, in order to further ensure clear real-time monitoring at night, the present application adds a fill light unit, and the image acquisition unit judges the wavelength of the received light. The proportion of invisible light is large at night, so the proportion of invisible light, that is, the received infrared light, is used to judge whether it is night, thereby controlling the operation of the fill light unit.

[0084] In a specific implementation process, the image acquisition unit may also send the received wavelength ratio to the MCU unit, and the MCU unit controls the power of the fill light unit according to the ratio of invisible light in the wavelength ratio.

[0085] For example, the PID control algorithm is used to adjust the brightness of the fill light unit according to the ambient light intensity. The PID controller in the MCU unit adjusts the PWM signal sent to the fill light unit through three parameters: proportional (P), integral (I) and differential (D), thereby controlling the brightness of the light. The duty cycle of the PWM signal determines the brightness of the light. The higher the duty cycle, the brighter the light.

[0086] A mapping function can also be designed to establish a corresponding relationship between the invisible light ratio and the light brightness. For example, when the invisible light ratio is higher than a certain threshold, the light brightness (PWM signal duty cycle) is increased by a fixed amount; when the invisible light ratio is lower than another threshold, the light brightness is reduced by a fixed amount.

[0087] Furthermore, in order to avoid sudden changes in light brightness causing exposure of individual frames in monitoring, a step-by-step brightness adjustment algorithm can be used to gradually adjust the light brightness until the target brightness is reached.

[0088] The MCU unit can also monitor the wavelength ratio fed back by the image acquisition unit in real time while adjusting the light brightness of the fill light unit, and adjust the light brightness according to the real-time wavelength ratio to maintain a constant wavelength ratio.

[0089] Furthermore, the cat's eye monitoring module may also include an audio input unit and an audio output unit, which are respectively used to collect environmental audio and output the audio sent from the server module and input by the mobile terminal to realize calls.

[0090] The above-mentioned embodiment can realize real-time monitoring of ambient light and adjust the fill light unit accordingly, thereby ensuring the clarity of nighttime monitoring and indirectly improving the security of the entire door lock system.

[0091] In some embodiments, the image acquisition unit includes:

[0092] The panoramic cat-eye lens, dual-channel filter and CIS unit are connected in sequence.

[0093] Among them, CIS stands for CMOS Image Sensor (Complementary Metal Oxide Semiconductor Image Sensor), which is the core component of the camera module. It is mainly responsible for converting the received light signal into an electrical signal, and then outputting a digital signal through analog-to-digital conversion for subsequent image processing (i.e., MCU unit).

[0094] CIS has a high degree of integration, and its structure mainly includes modules such as photosensitive area array (pixel array), timing control, analog signal processing, and analog-to-digital conversion. In the lens module, the role of CIS is to capture photons focused through the lens and convert them into electronic signals. These electronic signals are then converted into digital signals to form cat-eye video data. The performance of CIS directly affects the imaging quality of the camera, including key parameters such as resolution, color reproduction, and sensitivity.

[0095] A panoramic cat's eye lens is a cat's eye lens with a 180-degree vertical field of view and a 180-degree left and right field of view. It is used to collect ambient light and send it to a dual-channel filter. The center of the cat's eye lens coincides with the optical center of the CIS unit.

[0096] The dual-channel filter is used to transmit visible light with a wavelength of 400nm-650nm and infrared light with a wavelength of 900nm-1000nm to the CIS unit; the CIS unit generates cat-eye video data based on the received light waves.

[0097] In existing cat-eye camera modules, IRCUT components, namely infrared cut-off filters, are usually used. They are components used in cameras to improve image quality. Their function is to optimize the imaging effect of the camera by automatically switching different filters under different lighting conditions.

[0098] When there is sufficient light during the day, IRCUT will use an infrared cut-off filter to block infrared light from entering the camera, because infrared light will cause color cast in the image, making the picture appear unnatural red or purple. This will allow the camera to capture true color images.

[0099] At night or in dimly lit environments, IRCUT will automatically switch to a full-spectrum filter, allowing all bands of light (including infrared) to pass through, thereby enhancing the camera's night vision capability and improving imaging effects in low light conditions.

[0100] IRCUT components usually consist of the following parts:

[0101] Filters: Includes an infrared cutoff filter and a full spectrum filter.

[0102] Power part: It can be an electromagnetic, motor or other power source, used to drive the switching of the filter.

[0103] Control circuit: responsible for detecting ambient light and sending signals to the power part to achieve automatic switching of filters.

[0104] The above structure means that the size of the IRCUT component will be relatively large, resulting in a large size and high weight of the corresponding door lock device, and the IRCUT component is also very susceptible to vibration and damage.

[0105] In order to reduce the space required for structural installation so that it can be integrated with the door lock, and to avoid damage to the traditional IRCUT components caused by vibration when opening and closing the door, this application needs to remove the IRCUT components and introduce a dual-pass filter and a fill light unit to meet the needs of collecting images both during the day and at night.

[0106] In order to meet the demand for a 2-megapixel panoramic camera with a focal length of 50 cm and a field of view (FOV) greater than 180 degrees, it is necessary to select a CIS with an effective acquisition area aspect ratio of 1:1 and a maximum resolution of >2 million pixels, and a fisheye lens with a diagonal angle FOV (D) >180 degrees in the field of view when the lens is fixed at 50 cm.

[0107] Therefore, this application uses a cat-eye lens containing a double-pass filter for visible light (400nm~650nm) and infrared light (900~1000nm), and is matched with an infrared fill light. When the image acquisition unit works during the day, the visible light in the environment outside the door can be collected by the CIS unit through the lens. At this time, according to the wavelength ratio, the fill light unit is not lit; the panoramic cat-eye lens works in the RGB image acquisition mode; when the image acquisition unit works at night, the infrared light in the environment can be collected by the CIS unit through the panoramic cat-eye lens. At this time, according to the wavelength ratio, the fill light unit is lit to increase the ambient brightness, and the panoramic cat-eye lens works in the IR image acquisition mode. The RGB and IR modes can be switched adaptively according to the ambient illumination.

[0108] Compared with the traditional cat-eye camera lens with IRCUT components, the present application removes the IRCUT control components, reduces the installation space required for the structure, and can avoid damage to the IRCUT components caused by vibration caused by long-term opening and closing of the door during the use of the door lock, thereby improving the freedom of structural adaptation and meeting the reliability requirements of the door lock.

[0109] In some embodiments, the fill light unit is an infrared fill light with a wavelength of 940 nm; the transmittance of the dual-channel filter to infrared light with a wavelength of 900 nm-1000 nm is 45%.

[0110] Specifically, although the use of dual-channel filters can avoid vibration damage and reduce installation space, it also needs to face the serious color cast problem that objectively exists in dual-channel filters.

[0111] See also Figure 3By designing the spectrum of the dual-channel filter, when the image acquisition unit works during the day, a large amount of visible light with a wavelength of 400nm to 650nm is received by the CIS unit through the dual-channel filter, while when the image acquisition unit works at night, a large amount of invisible light with a wavelength of 900nm to 1000nm is received by the CIS unit through the filter. The reason for the color cast is that the visible light and invisible light (the invisible light here mainly refers to infrared light) in the environment are collected by the CIS unit at the same time, resulting in the superposition of infrared light in the color picture and a reddish color. The imaging principle is that the CIS unit receives the light reflected (refracted) by the object, and the relationship between the object reflectivity (refractive index) and the wavelength is that the longer the wavelength, the smaller the reflectivity (refractive index). When there are scenes of visible light and invisible light in the environment, because the refractive index of the 940nm fill light is small and the transmittance of the filter at 900nm to 1000nm is designed to be about 45%, the amount of infrared light collected by the CIS unit is small at this time, which can reduce the color cast problem.

[0112] It is worth noting that if the transmittance of the dual-channel filter for invisible light is adjusted in order to solve the color cast problem, the corresponding preset night threshold for judging day and night should also be appropriately lowered to ensure accurate judgment.

[0113] See also Figure 4 In some embodiments, the mobile terminal includes:

[0114] The correction unit is used to perform distortion correction on each image frame to obtain a panoramic rectangular image corresponding to each image frame.

[0115] The cutting and scaling unit is used to segment the panoramic rectangular image according to the preset cutting ratio in the split-screen display instruction to obtain multiple groups of video stream data of different sizes; enlarge or reduce each video stream data according to the preset scaling ratio in the split-screen display instruction, and place it at a preset display position.

[0116] Among them, the split-screen display instructions include: upper and lower split-screen instructions and left and right split-screen instructions.

[0117] The preset cutting ratio of the top and bottom split screen instruction is 70% of the upper side and 30% of the lower side of the panoramic rectangular image.

[0118] The preset cutting ratio of the left and right split screen instructions is 70% on the left and 50% on the right of the panoramic rectangular image.

[0119] Specifically, after obtaining the corrected panoramic rectangular image, for the upper and lower split-screen instructions, OpenglES can be used to define quadrilaterals of different sizes, with the upper screen as the large screen and the lower screen as the small screen, and the sizes of the large screen and the small screen are the preset scaling ratios; then the picture is cropped, with the upper 70% of the picture area mapped to the large screen; the lower 30% of the picture area mapped to the small screen. The small screen is an independent picture, which can facilitate users to observe and pay attention to the bottom of the lock.

[0120] As for the left-right split screen command, OpenglES is still used to draw quadrilaterals of different sizes on the left and right. The large screen is used as the main screen, and the small screen in the lower right corner is used as the secondary screen. The sizes of the main screen and the secondary screen are the preset scaling ratios. Then the screen is cropped, and the left 70% of the screen area is mapped to the main screen; the right 50% of the screen area is mapped to the secondary screen.

[0121] Furthermore, an operation of clicking the secondary screen to switch between the main screen and the secondary screen can be added, that is, when it is detected that the user's click area is within the range of the secondary screen, the zoom ratios and display areas corresponding to the video stream data of the two can be swapped.

[0122] The format conversion unit is used to convert the enlarged or reduced video stream data into RGB video data, and render and display the RGB video data at a preset display position.

[0123] Specifically, the original format of the video stream data is YUV, and the format conversion formula is:

[0124] R=Y+1.402*V

[0125] G=Y-0.344*U-0.714*V

[0126] B=Y+1.770*U

[0127] The top-bottom split screen mode and the left-right split screen mode provided in the above embodiments can provide corresponding emphasis display functions for users who focus on different areas, that is, the focused area is captured and enlarged separately to facilitate user viewing.

[0128] In some embodiments, the correction unit is specifically used to:

[0129] The circular image frame is converted into a rectangular image using an equidistant cylindrical projection model, and the pixel values ​​of the image frame are mapped into the rectangular image using a bilinear algorithm or a bicubic interpolation algorithm to obtain a panoramic rectangular image.

[0130] See also Figure 5 (Briefing Figure 3), the imaging principle of the cat-eye lens will cause distortion of the video edges, especially when the field of view is expanded to 180 degrees, the video edges will cause serious distortion; for example, the edge of the video will appear stretched and deformed, which cannot reflect the authenticity of the object and seriously affects the user's monitoring experience.

[0131] See also Figure 6 and Figure 7 , first map the cat-eye lens imaging to a spherical model, and then based on Figure 8 The equidistant cylindrical projection model shown expands the hemisphere into a rectangle to obtain a rectangular image; finally, through interpolation algorithms such as bilinear or bicubic, the pixel values ​​of each longitude and latitude in the cat's eye image frame can be mapped to the pixel values ​​of each corresponding pixel point in the rectangular image.

[0132] The above-mentioned embodiment adopts the equidistant cylindrical projection model for distortion correction, which can make the distortion correction algorithm relatively simple and improve the speed of video processing, especially when split-screen operation is required later, to ensure the real-time performance of the monitoring video; in addition, in the equidistant cylindrical projection model, the projections along the meridians and standard parallels are equidistant, which helps to reduce image distortion, especially near the standard parallels, thereby better maintaining the authenticity of the image.

[0133] In some embodiments, the mobile terminal further includes:

[0134] A VR display unit is used to receive VR display instructions, map cat's eye video data to a hemispherical surface; determine a viewpoint, a perspective projection matrix and a viewing angle transformation matrix within the hemispherical surface; obtain a first distance and a second distance based on a user's swiping of the viewpoint; determine a first transformation angle and a second transformation angle according to the first distance, the second distance and a screen density of the mobile terminal; determine a target display range according to the first transformation angle, the second transformation angle, the perspective projection matrix and the viewing angle transformation matrix, and render and display the cat's eye video data in the target display range.

[0135] Specifically, see Fig. 9 First, design a hemisphere with radius r=8 and the negative direction of the z-axis as the pole. Divide the surface of the sphere into 150 longitudes and 75 latitudes. The calculation formulas of polar angle and azimuth are as follows:

[0136]

[0137]

[0138] z=-r cosθ

[0139] Then, the coordinates P (x, y, z) of the hemispherical surface are calculated, and the texture coordinates of the image frame of the corresponding cat's eye video data are calculated, so that the original image frame is mapped to the inside of the hemispherical surface.

[0140] Then, set the viewpoint inside the sphere: take z=-1f as the observation point and the negative direction of the z axis as the observation direction.

[0141] Set the field of view inside the sphere: that is, set the perspective projection matrix, the bottom, top, left and right sides of the view volume are all 0.75f, the near clipping plane is 0.7f, and the far clipping plane is 10f. Among them, f is the focal length of the panoramic cat-eye lens.

[0142] The viewing angle transformation matrix is ​​set inside the sphere, with the pole as the viewing direction (the negative direction of the z-axis). When the user swipes the screen on the display of the mobile terminal, assuming that the sliding distances of the x-axis and y-axis are distanceX and distanceY respectively, the formula is:

[0143] deltaX = distanceX / screen density * 0.2

[0144] deltaY=distanceY / screen density*0.2

[0145] Wherein, distanceX and distanceY are respectively the first distance and the second distance, deltaX and deltaY are respectively the first transformation angle and the second transformation angle, and the screen density can be known according to the system parameters of the mobile terminal.

[0146] Specifically, after obtaining the first transformation angle and the second transformation angle, the perspective transformation matrix is ​​updated according to the first transformation angle and the second transformation angle. The perspective transformation matrix usually includes the direction of the user's head, so when the user swipes the screen, this matrix needs to be updated accordingly to reflect the new line of sight direction.

[0147] The perspective projection matrix is ​​responsible for converting a 3D scene into a 2D image to simulate the perspective effect of the human eye. This matrix depends on the field of view (FOV), aspect ratio, and near (zNear) and far (zFar) clipping planes.

[0148] Combining the updated view transformation matrix and perspective projection matrix, the rendering engine can calculate a new view frustum, which defines the part of the scene that should be rendered from the user's current perspective, i.e., the target display range.

[0149] Finally, the 3D scene is cropped according to the viewing frustum (i.e., the image frame mapped to the hemisphere), only the part within the viewing frustum is rendered, and the rendering result is output to the screen of the mobile terminal.

[0150] The above mentioned is the process of determining the image frame. Since the video is composed of multiple image frames, the above process is performed for each image frame to obtain the surveillance video displayed by VR.

[0151] See also Fig.10 Another embodiment of the present application provides a video display method for a cat's eye smart door lock, which is applied to a mobile terminal and may specifically include the following steps:

[0152] Step S1, receiving cat's eye video data; performing distortion correction on each image frame of the cat's eye video data to obtain a panoramic rectangular image corresponding to each image frame.

[0153] Specifically, for the distortion correction of cat's eye video data, a command trigger may be set separately, that is, the image frame is corrected when the correction command is received, otherwise the original circular cat's eye video data is still displayed.

[0154] Step S2, receiving a split-screen display instruction; cutting and scaling each panoramic rectangular image according to the split-screen display instruction to obtain and display multiple groups of video stream data of different sizes.

[0155] In some embodiments, the above-mentioned distortion correction is performed on each image frame of the cat's eye video data to obtain a panoramic rectangular image corresponding to each image frame, including:

[0156] The circular image frame is converted into a rectangular image by using an equidistant cylindrical projection model; the pixel values ​​of the image frame are mapped into the rectangular image by using a bilinear algorithm or a bicubic interpolation algorithm to obtain a panoramic rectangular image.

[0157] See also Figure 5 , because the imaging principle of the cat-eye lens will cause distortion of the video edges, especially when the field of view is expanded to 180 degrees, the video edges will cause serious distortion; for example, the edge of the video will appear stretched and deformed, which cannot reflect the authenticity of things and seriously affects the user's monitoring experience.

[0158] See also Figure 6 and Figure 7 , first map the cat-eye lens imaging to a spherical model, and then based on Figure 8 The equidistant cylindrical projection model shown expands the hemisphere into a rectangle to obtain a rectangular image; finally, through interpolation algorithms such as bilinear or bicubic, the pixel values ​​of each longitude and latitude in the cat's eye image frame can be mapped to the pixel values ​​of each corresponding pixel point in the rectangular image.

[0159] The above-mentioned embodiment adopts the equidistant cylindrical projection model for distortion correction, which can make the distortion correction algorithm relatively simple and improve the speed of video processing, especially when split-screen operation is required later, to ensure the real-time performance of the monitoring video; in addition, in the equidistant cylindrical projection model, the projections along the meridians and standard parallels are equidistant, which helps to reduce image distortion, especially near the standard parallels, thereby better maintaining the authenticity of the image.

[0160] In some embodiments, the above-mentioned cutting and scaling of each panoramic rectangular image according to the split-screen display instruction to obtain multiple groups of video stream data of different sizes include:

[0161] Step S211, cutting the upper 70% of the panoramic rectangular image to obtain a first split-screen image.

[0162] Step S212, cutting the lower 30% of the panoramic rectangular image to obtain a second split-screen image.

[0163] Step S213, each first split screen image is composed into a first video stream data, and each second split screen image is composed into a second video stream data. It can be understood that each image frame in the cat's eye video data is correspondingly cut into a plurality of first split screen images and second split screen images after correction, and each first split screen image is composed of the first video stream data in the order of the corresponding image frames, and each second split screen image is composed of the second video stream data in the order of the corresponding image frames.

[0164] Step S214, enlarge the first video stream data to the first display size, and reduce the second video stream data to the second display size. Specifically, after obtaining the corrected panoramic rectangular image, when the split-screen display instruction is an upper and lower split-screen instruction, a quadrilateral of different sizes can be defined through OpenglES, with the upper screen as the large screen and the lower screen as the small screen, and the sizes of the large screen and the small screen being the first display size and the second display size; then the picture is cropped, with the upper 70% of the picture area mapped to the large screen; and the lower 30% of the picture area mapped to the small screen. The small screen is an independent picture, which can facilitate users to observe and pay attention to the bottom of the lock.

[0165] Step S215: converting the data formats of the first video stream data and the second video stream data into RGB format.

[0166] Specifically, the original format of the video stream data is YUV, and the format conversion formula is:

[0167] R=Y+1.402*V

[0168] G=Y-0.344*U-0.714*V

[0169] B=Y+1.770*U

[0170] The upper and lower split screen mode provided in the above embodiment can provide corresponding emphasis display functions for users who focus on different areas, that is, the focused area is captured and enlarged separately to facilitate user viewing.

[0171] In some embodiments, the above-mentioned cutting and scaling of each panoramic rectangular image according to the split-screen display instruction to obtain multiple groups of video stream data of different sizes include:

[0172] Step S221, cutting the left 70% of the panoramic rectangular image to obtain a first split-screen image.

[0173] Step S222, cutting the right 50% of the panoramic rectangular image to obtain a second split-screen image.

[0174] In actual applications, the left-right or top-bottom ratio can be freely adjusted by developers.

[0175] Step S223, each first split-screen image is combined into a first video stream data, and each second split-screen image is combined into a second video stream data; it can be understood that after correction, each image frame in the cat's eye video data is correspondingly cut into multiple first split-screen images and second split-screen images, and each first split-screen image is combined into the first video stream data in the order of the corresponding image frames, and each second split-screen image is combined into the second video stream data in the order of the corresponding image frames.

[0176] Step S224, enlarge the first video stream data to the first display size, and reduce the second video stream data to the second display size; when the split screen display instruction is a left-right split screen instruction, OpenglES is still used to draw quadrilaterals of different sizes on the left and right, with the large screen as the main screen and the small screen in the lower right corner as the secondary screen. The sizes of the main screen and the secondary screen are the first display size and the second display size. Then the picture is cropped, and the left 70% of the picture area is mapped to the main screen; the right 50% of the picture area is mapped to the secondary screen.

[0177] Furthermore, an operation of clicking the secondary screen to switch between the main screen and the secondary screen can be added, that is, when it is detected that the user's click area is within the range of the secondary screen, the zoom ratios and display areas corresponding to the video stream data of the two can be swapped.

[0178] Step S225: converting the data formats of the first video stream data and the second video stream data into RGB format.

[0179] Specifically, the original format of the video stream data is YUV, and the format conversion formula is:

[0180] R=Y+1.402*V

[0181] G=Y-0.344*U-0.714*V

[0182] B=Y+1.770*U

[0183] The left-right split screen mode provided in the above embodiment can provide corresponding focus display functions for users who focus on different areas, that is, the focused area is captured and enlarged separately to facilitate user viewing.

[0184] See also Fig.11In some embodiments, the method further comprises:

[0185] Step S31, receiving a VR display instruction, and mapping the cat's eye video data to a hemispherical surface.

[0186] Step S32, determining the viewpoint, perspective projection matrix and viewing angle transformation matrix within the hemisphere.

[0187] Step S33, obtaining a first distance and a second distance based on the user's swiping of the viewpoint.

[0188] Step S34, determining a first transformation angle and a second transformation angle according to the first distance, the second distance and the screen density.

[0189] Step S35, determining a target display range according to the first transformation angle, the second transformation angle, the perspective projection matrix and the viewing angle transformation matrix, and rendering and displaying the cat's eye video data in the target display range.

[0190] Specifically, see Fig. 9 First, design a hemisphere with radius r=8 and the negative direction of the z-axis as the pole. Divide the surface of the sphere into 150 longitudes and 75 latitudes. The calculation formulas of polar angle and azimuth are as follows:

[0191]

[0192] z=-r cosθ

[0193] Then, the coordinates P (x, y, z) of the hemispherical surface are calculated, and the texture coordinates of the image frame of the corresponding cat's eye video data are calculated, so that the original image frame is mapped to the inside of the hemispherical surface.

[0194] Then, set the viewpoint inside the sphere: take z=-1f as the observation point and the negative direction of the z axis as the observation direction.

[0195] Set the field of view inside the sphere: that is, set the perspective projection matrix, the bottom, top, left and right sides of the view volume are all 0.75f, the near clipping plane is 0.7f, and the far clipping plane is 10f. Among them, f is the focal length of the panoramic cat-eye lens.

[0196] The viewing angle transformation matrix is ​​set inside the sphere, with the pole as the viewing direction (the negative direction of the z-axis). When the user swipes the screen on the display of the mobile terminal, assuming that the sliding distances of the x-axis and y-axis are distanceX and distanceY respectively, the formula is:

[0197] deltaX = distanceX / screen density * 0.2

[0198] deltaY=distanceY / screen density*0.2

[0199] Wherein, distanceX and distanceY are respectively the first distance and the second distance, deltaX and deltaY are respectively the first transformation angle and the second transformation angle, and the screen density can be known according to the system parameters of the mobile terminal.

[0200] Specifically, after obtaining the first transformation angle and the second transformation angle, the perspective transformation matrix is ​​updated according to the first transformation angle and the second transformation angle. The perspective transformation matrix usually includes the direction of the user's head, so when the user swipes the screen, this matrix needs to be updated accordingly to reflect the new line of sight direction.

[0201] The perspective projection matrix is ​​responsible for converting a 3D scene into a 2D image to simulate the perspective effect of the human eye. This matrix depends on the field of view (FOV), aspect ratio, and near (zNear) and far (zFar) clipping planes.

[0202] Combining the updated view transformation matrix and perspective projection matrix, the rendering engine can calculate a new view frustum, which defines the part of the scene that should be rendered from the user's current perspective, i.e., the target display range.

[0203] Finally, the 3D scene is cropped according to the viewing cone (i.e., the image frame mapped to the hemisphere), only the part within the viewing cone is rendered, and the rendering result is output to the screen of the mobile terminal.

[0204] The above mentioned is the process of determining the image frame. Since the video is composed of multiple image frames, the above process is performed for each image frame to obtain the surveillance video displayed by VR.

[0205] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0206] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A cat-eye intelligent door lock system, characterized in that: include: Cat's eye monitoring module, server module and mobile terminal; The cat's eye monitoring module is integrated in the door lock module, and is used to obtain cat's eye video data and send it to the server module; The server module is used to send the received cat's eye video data to the mobile terminal; The mobile terminal is used to perform distortion correction on each image frame of the cat's eye video data to obtain a panoramic rectangular image corresponding to each image frame; receive a split-screen display instruction, cut and scale each panoramic rectangular image according to the split-screen display instruction, obtain and display multiple groups of video stream data of different sizes.

2. The cat-eye intelligent door lock system according to claim 1, characterized in that: The cat's eye monitoring module includes: An image acquisition unit, an MCU unit and a fill light unit connected in sequence; The image acquisition unit is used to collect the cat's eye video data and send it to the MCU unit; and determine whether the received infrared light ratio reaches a preset night threshold; if so, send a mode switching instruction to the MCU unit; The MCU unit is used to send the received cat-eye video data to the server module; and, when receiving the mode switching instruction, control the fill light unit to light up.

3. The cat-eye intelligent door lock system according to claim 2, characterized in that: The image acquisition unit comprises: The panoramic cat-eye lens, dual-channel filter and CIS unit are connected in sequence; The dual-channel filter is used to transmit visible light with a wavelength of 400nm-650nm and infrared light with a wavelength of 900nm-1000nm to the CIS unit; the CIS unit generates the cat's eye video data based on the received light waves.

4. The cat-eye intelligent door lock system according to claim 3, characterized in that: The fill light unit is an infrared fill light with a wavelength of 940nm; the transmittance of the dual-channel filter to infrared light with a wavelength of 900nm-1000nm is 45%.

5. The cat-eye intelligent door lock system according to claim 1, characterized in that: The mobile terminal comprises: A correction unit, used for performing distortion correction on each of the image frames to obtain a panoramic rectangular image corresponding to each of the image frames; A cutting and scaling unit, configured to segment the panoramic rectangular image according to a preset cutting ratio in the split-screen display instruction to obtain a plurality of groups of video stream data of different sizes; to enlarge or reduce each of the video stream data according to the preset scaling ratio in the split-screen display instruction, and to place the video stream data at a preset display position; The format conversion unit is used to convert the enlarged or reduced video stream data into RGB video data, and render and display the RGB video data at the preset display position.

6. The cat-eye intelligent door lock system according to claim 5, characterized in that: The correction unit is specifically used for: The circular image frame is converted into a rectangular image by adopting an equidistant cylindrical projection model, and the pixel values ​​of the image frame are mapped into the rectangular image by a bilinear algorithm or a bicubic interpolation algorithm to obtain the panoramic rectangular image.

7. The cat-eye intelligent door lock system according to claim 6, characterized in that: The mobile terminal further includes: A VR display unit is used to receive a VR display instruction, map the cat's eye video data to a hemispherical surface; determine a viewpoint, a perspective projection matrix and a viewing angle transformation matrix within the hemispherical surface; obtain a first distance and a second distance of a user's swiping based on the viewpoint; determine a first transformation angle and a second transformation angle according to the first distance, the second distance and the screen density of the mobile terminal; determine a target display range according to the first transformation angle, the second transformation angle, the perspective projection matrix and the viewing angle transformation matrix, and render and display the cat's eye video data in the target display range.

8. The cat-eye intelligent door lock system according to claim 5, characterized in that: The split-screen display instruction includes: Up and down split screen instructions and left and right split screen instructions; The preset cutting ratio of the upper and lower split screen instruction is 70% of the upper side and 30% of the lower side of the panoramic rectangular image; The preset cutting ratio of the left-right split screen instruction is 70% on the left side and 50% on the right side of the panoramic rectangular image.

9. A video display method for a cat-eye smart door lock, characterized in that: Applied to mobile terminals, including: Receive cat's eye video data; perform distortion correction on each image frame of the cat's eye video data to obtain a panoramic rectangular image corresponding to each image frame; Receive a split-screen display instruction; cut and scale each of the panoramic rectangular images according to the split-screen display instruction to obtain and display multiple groups of video stream data of different sizes.

10. The video display method of the cat's eye smart door lock according to claim 9, characterized in that: The step of performing distortion correction on each image frame of the cat's eye video data to obtain a panoramic rectangular image corresponding to each image frame includes: The circular image frame is converted into a rectangular image by adopting an equidistant cylindrical projection model; the pixel values ​​of the image frame are mapped into the rectangular image by a bilinear algorithm or a bicubic interpolation algorithm to obtain the panoramic rectangular image.

11. The video display method of the cat's eye smart door lock according to claim 9, characterized in that: The step of cutting and scaling each of the panoramic rectangular images according to the split-screen display instruction to obtain multiple groups of video stream data of different sizes includes: Cutting the upper 70% of the panoramic rectangular image to obtain a first split-screen image; Cutting the lower 30% of the panoramic rectangular image to obtain a second split-screen image; Combining the first split-screen images into first video stream data, and combining the second split-screen images into second video stream data; Enlarging the first video stream data to a first display size, and reducing the second video stream data to a second display size; The data formats of the first video stream data and the second video stream data are both converted into RGB format.

12. The video display method of cat-eye smart door lock according to claim 9, characterized in that: The step of cutting and scaling each of the panoramic rectangular images according to the split-screen display instruction to obtain multiple groups of video stream data of different sizes includes: Cutting the left 70% of the panoramic rectangular image to obtain a first split-screen image; Cutting the right 50% of the panoramic rectangular image to obtain a second split-screen image; Combining the first split-screen images into first video stream data, and combining the second split-screen images into second video stream data; Enlarging the first video stream data to a first display size, and reducing the second video stream data to a second display size; The data formats of the first video stream data and the second video stream data are both converted into RGB format.

13. The video display method of the cat's eye smart door lock according to claim 9, characterized in that: Also includes: Receiving a VR display instruction, and mapping the cat's eye video data to a hemispherical surface; Determining a viewpoint, a perspective projection matrix, and a viewing angle transformation matrix within the hemisphere; Acquire a first distance and a second distance of the user's swiping based on the viewpoint; determining a first transformation angle and a second transformation angle according to the first distance, the second distance and a screen density; A target display range is determined according to the first transformation angle, the second transformation angle, the perspective projection matrix and the viewing angle transformation matrix, and the cat's eye video data in the target display range is rendered and displayed.

Images