A physical whiteboard perspective method and a virtual whiteboard generation method

Abstract

The present invention provides a physical whiteboard perspective method and a virtual whiteboard generation method. By using the Hoffman line detection method and counting the number of overlapping lines, the judgment dimension of the whiteboard-related lines is increased, the accuracy of line extraction is improved, and then the accuracy of virtual whiteboard acquisition is improved, and the position of the whiteboard is better identified. On the basis of generating a high-precision virtual whiteboard, the purity of the whiteboard color is improved by color enhancement processing, and the virtual whiteboard corresponding to each frame of the physical whiteboard image is processed according to a preset algorithm to obtain a background image, a motion image and a color difference image, and then a foreground mask is obtained. The foreground mask of the current frame, the color enhancement picture of the current frame and the full perspective picture of the previous frame are used to achieve a smooth perspective effect of the character. The present invention not only reduces the computational complexity of the virtual whiteboard generation algorithm and the whiteboard perspective algorithm, but also improves the accuracy of detection and perspective, so that users can get a better experience.

Description

Technical Field

[0001] The present invention relates to the field of image recognition, and in particular to a physical whiteboard perspective method and a virtual whiteboard generation method. Background Art

[0002] The existing technical solution includes a method for whiteboard detection and perspective. The method first captures an image through a camera, then extracts the edge information of the whiteboard based on the image, thereby locating the specific position of the whiteboard, and correcting the whiteboard image by using affine transformation on the whiteboard; further, foreground and background detection is performed based on the color information between adjacent frames, and a multi-frame fusion strategy is used to achieve character perspective and whiteboard color purification.

[0003] When implementing the present invention, the inventors found that the prior art has the following disadvantages: dense color blocks in the whiteboard are unstable and will constantly change; when there are too many lines in the physical whiteboard, the corner point detection of the whiteboard is prone to false detection, resulting in the inability to perform correct affine transformation on the whiteboard; whiteboard detection and perspective are overall time-consuming and have high requirements on hardware performance.

[0004] Therefore, a whiteboard perspective strategy is urgently needed to solve the problems of low efficiency and high cost of existing whiteboard perspective. Summary of the invention

[0005] The embodiments of the present invention provide a physical whiteboard perspective method and a virtual whiteboard generation method to improve the whiteboard perspective efficiency.

[0006] In order to solve the above problem, an embodiment of the present invention provides a physical whiteboard perspective method, including:

[0007] Acquire virtual images of multiple frames of physical whiteboard images, and perform preset color enhancement processing on the virtual whiteboard of each frame of the physical whiteboard image to obtain an enhanced image of each frame of the physical whiteboard image;

[0008] In the enhanced image of each frame of the physical whiteboard image, according to the enhanced image of the virtual whiteboard of the current frame and the enhanced image of the virtual whiteboard of the previous frame, in combination with a preset weighting algorithm, a motion image of each frame of the physical whiteboard image is obtained; according to the enhanced image of the virtual whiteboard image of the current frame and the background image of the virtual whiteboard of the first frame, in combination with a preset weighting algorithm, a color difference image of each frame of the physical whiteboard image is obtained; wherein the background image of the virtual whiteboard of the first frame is extracted according to a preset algorithm to obtain the background image of the virtual whiteboard of the physical whiteboard of the first frame;

[0009] According to the motion map and the color difference map, a foreground mask of each frame of the physical whiteboard image is generated by using a median filter and a dilation algorithm;

[0010] According to the foreground mask of the current frame physical whiteboard image, the enhanced image of the current frame physical whiteboard image, and the full perspective image of the previous frame physical whiteboard image, a semi-perspective image of the current frame physical whiteboard image is obtained by weighted fusion with preset coefficients.

[0011] As can be seen from the above, the present invention has the following beneficial effects: on the basis of generating a highly accurate virtual whiteboard, the purity of the whiteboard color is improved by color enhancement processing, and the virtual whiteboard corresponding to each frame of the physical whiteboard image is processed according to a preset algorithm to obtain a background image, a motion image and a color difference image, and then a foreground mask is obtained. The effect of smooth perspective of the character is achieved through the foreground mask of the current frame, the color enhanced image of the current frame and the full perspective image of the previous frame. The present invention not only reduces the computational complexity of the whiteboard perspective algorithm, but also improves the accuracy of detection and perspective, so that users can get a better experience.

[0012] As an improvement of the above scheme, the background color image of the first frame of the virtual whiteboard is specifically as follows: according to the virtual whiteboard of each frame of the physical whiteboard image, the Y component of the virtual whiteboard of the first frame of the physical whiteboard image is divided into sliding windows of a preset size to obtain multiple blocks; the multiple blocks are sorted by maximum heap, and the mean of the Y component values ​​corresponding to the blocks sorted in a preset range is extracted to obtain the background color image of the first frame of the virtual whiteboard. By using the local maximum mean based on the sliding window, the efficiency of the whiteboard background color extraction is improved, thereby obtaining a high-quality whiteboard background color.

[0013] As an improvement of the above scheme, the virtual images of multiple frames of physical whiteboard images are obtained, and the virtual whiteboard of each frame of the physical whiteboard image is subjected to preset color enhancement processing to obtain an enhanced image of each frame of the physical whiteboard image. Specifically, the background color of each frame of the physical whiteboard image is extracted according to the virtual whiteboard of each frame of the physical whiteboard image to obtain the initial pixel of each point in each frame of the physical whiteboard image; and the initial pixel of each point in each frame of the physical whiteboard image is color enhanced according to the preset pixel activation function. After the background color image of the whiteboard is extracted, the preset pixel activation function is used for color enhancement, which improves the purity of the background color image, thereby improving the efficiency of image fusion.

[0014] As an improvement of the above scheme, in the enhanced image of each frame of the physical whiteboard image, according to the enhanced image of the virtual whiteboard of the current frame, the enhanced image of the virtual whiteboard of the previous frame and the background image of the virtual whiteboard of the first frame, combined with a preset weighting algorithm, a motion image of each frame of the physical whiteboard image and a color difference image of each frame of the physical whiteboard image are obtained respectively, specifically:

[0015] Each frame of the enhanced image is reduced to one-fourth of the original image size, and the enhanced image of the virtual whiteboard of the current frame after reduction is directly subtracted from the enhanced image of the virtual whiteboard of the previous frame after reduction, and then weighted addition is performed to obtain the absolute value. When the pixel motion change is greater than the motion threshold, the value of the corresponding position of the motion map is increased by 1, otherwise it is cleared to zero. The formula is:

[0016]

[0017] Among them, C small (x) shows the YUV vector of the enhanced image of the virtual whiteboard of the current frame after reduction, P small (x) represents the YUV vector of the enhanced image of the previous frame of the virtual whiteboard after reduction, W YUV Indicates the weight of the YUV component, D thresh Represents the motion threshold; the enhanced image of the virtual whiteboard of the current frame after reduction is subtracted from the background image of the virtual whiteboard of the first frame after reduction, and then the weighted addition is performed to obtain the absolute value. When the chromaticity change is greater than the color difference threshold, the value of the corresponding position of the color difference map is 0, otherwise it is 1. The formula is:

[0018]

[0019] Among them, C small (x) represents the YUV vector of the reduced enhanced image of the virtual whiteboard of the current frame, and W small (x) represents the YUV vector of the background image of the first frame of the virtual whiteboard after reduction, W′ YUV Indicates the weight of the YUV component, H thresh Represents the color difference threshold; by generating a foreground mask for the enhanced image of the virtual whiteboard in the current frame, the enhanced image of the virtual whiteboard in the previous frame, and the background color image of the virtual whiteboard in the first frame, the accuracy of the whiteboard perspective is improved, laying a solid foundation for the perspective in subsequent steps.

[0020] As an improvement of the above scheme, the foreground mask of the current frame physical whiteboard image, the enhanced image of the current frame physical whiteboard image, and the full perspective image of the previous frame physical whiteboard image are weighted and fused with preset coefficients to obtain a semi-perspective image of the current frame physical whiteboard image. Specifically, according to the foreground mask of the current frame physical whiteboard image, the enhanced image of the current frame physical whiteboard image, and the full perspective image of the previous frame physical whiteboard image, the image fusion is performed in combination with preset coefficients to obtain the full perspective image of the current frame physical whiteboard. The formula is:

[0021]

[0022] Among them, B(x) is the full perspective image of the physical whiteboard in the current frame, A(x) is the full perspective image of the physical whiteboard in the previous frame, C(x) is the enhanced image of the physical whiteboard in the current frame, and M(x) is the foreground mask of the physical whiteboard in the current frame. According to the full perspective image of the physical whiteboard in the current frame and the enhanced image of the physical whiteboard in the current frame, the secondary image fusion is performed in combination with the preset coefficients to obtain the semi-perspective image of the physical whiteboard in the current frame. The formula is:

[0023]

[0024] Among them, B(x) is the full perspective image of the physical whiteboard in the current frame, C(x) is the enhanced image of the physical whiteboard in the current frame, and R(x) is the semi-perspective image of the physical whiteboard in the current frame. Combining the color difference map, motion map and foreground mask, through the full perspective image fusion and semi-perspective image fusion, the rules and layer fusion scheme of the foreground and background detection in the whiteboard are optimized, making the final processed whiteboard image smoother and realizing a more beautiful perspective function.

[0025] Accordingly, the present invention also provides a method for generating a virtual whiteboard, comprising:

[0026] Acquire a first frame of physical whiteboard image, and extract a set of edge points of the first frame of physical whiteboard image;

[0027] According to a preset Huffman line detection algorithm, the edge point set is mapped into a first line set, and the number of intersection overlaps of each line in the first line set is calculated to obtain a valid line set of the first frame of the physical whiteboard image; wherein the number of intersection overlaps of the lines in the valid line set is greater than a filtering threshold;

[0028] According to the valid straight line set of the first frame of the physical whiteboard image, a whiteboard area of ​​the first frame of the physical whiteboard image is obtained by combination;

[0029] Performing homography transformation processing on the whiteboard area to obtain a virtual whiteboard of the first frame of physical whiteboard image;

[0030] The virtual whiteboard of the first frame of the physical whiteboard can be used to determine the virtual whiteboards of the physical whiteboards of other frames, so that the physical whiteboard perspective method of the present invention is applied to perform whiteboard perspective on the virtual whiteboards of the physical whiteboard images of all frames.

[0031] As can be seen from the above, the present invention has the following beneficial effects: the present invention increases the judgment dimension of whiteboard-related straight lines by using the Hoffman line detection method and counting the number of overlapping lines, thereby improving the accuracy of straight line extraction, thereby improving the accuracy of virtual whiteboard acquisition, and better identifying the position of the whiteboard. In addition, compared with the prior art, the computational complexity of detecting the whiteboard to generate the virtual whiteboard is also reduced.

[0032] As an improvement of the above scheme, according to the preset Huffman line detection algorithm, the edge point set is mapped into a first line set, and the number of intersection overlaps of each line in the first line set is calculated to obtain a valid line set of the first frame physical whiteboard image, specifically: constructing a Huffman space, mapping the edge point set N into multiple straight line segments N H1 、N H2 ,…,N Hn , the function expression of each straight line is: Among them, θ∈[0°,360°], so as to obtain the first straight line set of the first frame physical whiteboard image; according to the preset overlap resolution value, calculate the intersection coordinates of the first straight line set, and obtain the overlap times of each intersection; in the first frame physical whiteboard image, map the valid intersections to valid straight lines in the Euler space, so as to obtain the valid straight line set; wherein, the valid intersections are intersections with the overlap times greater than the overlap threshold. The Huffman line detection algorithm is used to perform overlap detection on the straight lines mapped by each edge point. When the overlap resolution value is set, the intersections with the overlap times greater than the overlap threshold are mapped to valid straight lines in the Euler space, which ensures the extraction of high-quality straight lines in the picture and improves the accuracy and efficiency of whiteboard straight line extraction.

[0033] As an improvement of the above scheme, before the combination obtains the whiteboard area of ​​the first frame of the physical whiteboard image, it also includes: according to the number of overlaps of the overlapping straight lines in the effective straight line set, through a preset filtering algorithm, filtering the straight lines with a small number of overlaps and incorrect angles in the effective straight line set to obtain the whiteboard straight line set of the first frame of the physical whiteboard image. By retaining the number of overlaps and combining it with the preset filtering algorithm, a large number of falsely detected whiteboard straight lines are filtered out, and the accuracy of whiteboard straight line extraction is further improved in the case of floor boundaries and wall boundaries.

[0034] As an improvement of the above scheme, the combination obtains the whiteboard area of ​​the first frame of the physical whiteboard image, specifically: according to the whiteboard straight line set, the combination obtains the valid quadrilateral set of the first frame of the physical whiteboard image; wherein the valid quadrilateral is a quadrilateral whose area is greater than a preset area value; the number of overlaps of the quadrilaterals in the valid quadrilateral set is counted, and the area represented by the quadrilateral with the highest number of overlaps is selected as the whiteboard area of ​​the first frame of the physical whiteboard image. On the basis of extracting high-quality straight lines, the whiteboard area in the physical whiteboard image is obtained by determining the number of overlaps of the constituent quadrilaterals, and the accuracy of the determination of the whiteboard area is increased based on the overlap threshold judgment.

[0035] As an improvement of the above scheme, the whiteboard area is subjected to homography transformation processing to obtain a virtual whiteboard of the first frame of the physical whiteboard image, specifically: the four corner points of the whiteboard area are calculated, and the four coordinate points of the whiteboard area are obtained according to the camera model and the focal length of the camera; according to the four corner points of the whiteboard area and the four coordinate points of the whiteboard area, combined with memory optimization and assembly optimization, a high-performance homography transformation is performed in the YUV space, so as to obtain a virtual whiteboard of each frame of the physical whiteboard image. Through the calculation of corner points and coordinate points, the efficiency of homography transformation is improved on the basis of memory optimization and assembly optimization, so as to obtain a virtual whiteboard faster.

[0036] Correspondingly, the present invention also provides a physical whiteboard perspective device, comprising: an image enhancement module, a fusion module, a foreground mask module and a perspective module;

[0037] The image enhancement module is used to obtain virtual images of multiple frames of physical whiteboard images, and obtain an enhanced image of each frame of the physical whiteboard image after performing a preset color enhancement process on the virtual whiteboard of each frame of the physical whiteboard image;

[0038] The fusion module is used to obtain, in the enhanced image of each frame of the physical whiteboard image, a motion image of each frame of the physical whiteboard image according to the enhanced image of the virtual whiteboard of the current frame and the enhanced image of the virtual whiteboard of the previous frame, in combination with a preset weighting algorithm; obtain a color difference image of each frame of the physical whiteboard image according to the enhanced image of the virtual whiteboard of the current frame and the background image of the virtual whiteboard of the first frame, in combination with a preset weighting algorithm; wherein the background image of the virtual whiteboard of the first frame is extracted according to the preset algorithm to obtain the background image of the virtual whiteboard of the physical whiteboard of the first frame;

[0039] The foreground mask module is used to generate a foreground mask of each frame of the physical whiteboard image according to the motion map and the color difference map by using a median filter and a dilation algorithm;

[0040] The perspective module is used to obtain a semi-perspective image of the current frame physical whiteboard image by weighted fusion using preset coefficients according to the foreground mask of the current frame physical whiteboard image, the enhanced image of the current frame physical whiteboard image, and the full perspective image of the previous frame physical whiteboard image.

[0041] As an improvement of the above scheme, the background image of the first frame of the virtual whiteboard is specifically as follows: according to the virtual whiteboard of each frame of the physical whiteboard image, the Y component of the virtual whiteboard of the first frame of the physical whiteboard image is divided into sliding windows of a preset size to obtain multiple blocks; the multiple blocks are sorted by a maximum heap, and the mean of the Y component values ​​corresponding to the blocks sorted in a preset range are extracted to obtain the background image of the first frame of the virtual whiteboard.

[0042] As an improvement of the above solution, the image enhancement module includes: a first pixel unit and a second pixel unit;

[0043] The first pixel unit is used to extract the background color of each frame of the physical whiteboard image according to the virtual whiteboard of each frame of the physical whiteboard image, and obtain the initial pixel of each point in each frame of the physical whiteboard image;

[0044] The second pixel unit is used to perform color enhancement on the initial pixel of each point in each frame of the physical whiteboard image according to a preset pixel activation function.

[0045] As an improvement of the above solution, the fusion module includes: a motion image unit and a color difference image unit;

[0046] The motion map unit is used to reduce each frame of the enhanced image to one-fourth of the original image size, and directly perform weighted addition on the difference between the enhanced image of the virtual whiteboard of the current frame after reduction and the enhanced image of the virtual whiteboard of the previous frame after reduction to obtain the absolute value. When the pixel motion change is greater than the motion threshold, the value of the corresponding position of the motion map is increased by 1, otherwise it is cleared to zero. The formula is:

[0047]

[0048] Among them, C small (x) shows the YUV vector of the enhanced image of the virtual whiteboard of the current frame after reduction, P small (x) represents the YUV vector of the enhanced image of the previous frame of the virtual whiteboard after reduction, W YUV Indicates the weight of the YUV component, D thresh represents the motor threshold;

[0049] The color difference map unit is used to perform weighted addition after subtracting the enhanced image of the virtual whiteboard of the current frame after reduction and the background color image of the virtual whiteboard of the first frame after reduction to obtain an absolute value. When the chromaticity change is greater than the color difference threshold, the value of the corresponding position of the color difference map is 0, otherwise it is 1. The formula is:

[0050]

[0051] Among them, C small (x) represents the YUV vector of the reduced enhanced image of the virtual whiteboard of the current frame, and W small (x) represents the YUV vector of the background image of the first frame of the virtual whiteboard after reduction, W′ YUV Indicates the weight of the YUV component, H thresh Indicates the color difference threshold.

[0052] As an improvement of the above solution, the perspective module includes: a full perspective unit and a semi-perspective unit;

[0053] The full perspective unit is used to perform image fusion based on the foreground mask of the current frame physical whiteboard image, the enhanced image of the current frame physical whiteboard image, and the full perspective image of the previous frame physical whiteboard image in combination with a preset coefficient, so as to obtain the full perspective image of the current frame physical whiteboard. The formula is:

[0054]

[0055] Wherein, B(x) is the full perspective image of the physical whiteboard in the current frame, A(x) is the full perspective image of the physical whiteboard in the previous frame, C(x) is the enhanced image of the physical whiteboard in the current frame, and M(x) is the foreground mask of the physical whiteboard in the current frame.

[0056] The semi-perspective unit is used to perform secondary image fusion based on the full perspective image of the current frame physical whiteboard and the enhanced image of the current frame physical whiteboard image in combination with a preset coefficient, so as to obtain a semi-perspective image of the current frame physical whiteboard image. The formula is:

[0057]

[0058] Among them, B(x) is the full perspective image of the physical whiteboard in the current frame, C(x) is the enhanced image of the physical whiteboard in the current frame, and R(x) is the semi-perspective image of the physical whiteboard in the current frame.

[0059] Correspondingly, the present invention also provides a virtual whiteboard generation device, comprising: an edge point extraction module, a straight line extraction module, a combination module, a transformation module and a physical whiteboard perspective module;

[0060] The edge point extraction module is used to obtain a first frame of physical whiteboard image and extract an edge point set of the first frame of physical whiteboard image;

[0061] The straight line extraction module is used to map the edge point set into a first straight line set according to a preset Huffman line detection algorithm, and calculate the number of intersection overlaps of each straight line in the first straight line set to obtain a valid straight line set of the first frame of the physical whiteboard image; wherein the number of intersection overlaps of the straight lines in the valid straight line set is greater than a filtering threshold;

[0062] The combining module is used to combine and obtain the whiteboard area of ​​the first frame of the physical whiteboard image according to the valid straight line set of the first frame of the physical whiteboard image;

[0063] The transformation module is used to perform homography transformation processing on the whiteboard area to obtain a virtual whiteboard of the first frame of physical whiteboard image;

[0064] The physical whiteboard perspective module is used to determine the virtual whiteboards of the physical whiteboards of other frames based on the virtual whiteboard of the physical whiteboard of the first frame, so as to apply the physical whiteboard perspective method of the present invention to perform whiteboard perspective on the virtual whiteboards of the physical whiteboard images of all frames.

[0065] As an improvement of the above solution, the straight line extraction module includes: a first mapping unit, an overlapping unit and a second mapping unit;

[0066] The first mapping unit is used to construct a Huffman space to map the edge point set N into a plurality of straight lines N H1 、N H2 ,…,N Hn , the function expression of each straight line is: Wherein, θ∈[0°,360°], thereby obtaining a first straight line set of the first frame of the physical whiteboard image;

[0067] The overlapping unit is used to calculate the coordinates of the intersection points of the first set of straight lines according to a preset overlapping resolution value, and obtain the number of overlapping times of each intersection point;

[0068] The second mapping unit is used to map valid intersections in the first frame of the physical whiteboard image into valid straight lines in the Euler space, so as to obtain the valid straight line set; wherein the valid intersections are intersections whose overlap times are greater than an overlap threshold.

[0069] As an improvement of the above scheme, before the combination obtains the whiteboard area of ​​the first frame of the physical whiteboard image, it also includes: according to the number of overlaps of the overlapping straight lines in the valid straight line set, through a preset filtering algorithm, filtering the straight lines with a small number of overlaps and incorrect angles in the valid straight line set to obtain the whiteboard straight line set of the first frame of the physical whiteboard image.

[0070] As an improvement of the above solution, the combination module includes: a first combination unit and a second combination unit;

[0071] The first combination unit is used to combine the whiteboard straight line set to obtain a valid quadrilateral set of the first frame of the physical whiteboard image; wherein the valid quadrilateral is a quadrilateral whose area is greater than a preset area value;

[0072] The first combining unit is used to count the overlapping times of the quadrilaterals in the valid quadrilateral set, and select the area represented by the quadrilateral with the highest overlapping times as the whiteboard area of ​​the first frame of the physical whiteboard image.

[0073] As an improvement of the above solution, the transformation module includes: a first processing unit and a second processing unit;

[0074] The first processing unit is used to calculate four corner points of the whiteboard area, and obtain four coordinate points of the whiteboard area according to the camera model and the camera focal length;

[0075] The second processing unit is used to perform high-performance homography transformation in YUV space according to the four corner points of the whiteboard area and the four coordinate points of the whiteboard area, combined with memory optimization and assembly optimization, so as to obtain a virtual whiteboard of each frame of the physical whiteboard image.

[0076] Correspondingly, the present invention also provides a computer terminal device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, and when the processor executes the computer program, it implements a physical whiteboard perspective method and a virtual whiteboard generation method as described in the present invention.

[0077] Correspondingly, the present invention also provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute a physical whiteboard perspective method and a virtual whiteboard generation method as described in the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Figure 1 It is a flowchart of a physical whiteboard perspective method provided by an embodiment of the present invention;

[0079] Figure 2 is a schematic structural diagram of a physical whiteboard perspective device provided by an embodiment of the present invention;

[0080] Figure 3 is a flowchart of a method for generating a virtual whiteboard provided by an embodiment of the present invention;

[0081] Figure 4 is a structural schematic diagram of a device for generating a virtual whiteboard provided by an embodiment of the present invention;

[0082] Figure 5 is a flowchart of a method for generating a virtual whiteboard provided by another embodiment of the present invention;

[0083] Figure 6 It is a schematic diagram of a process of extracting whiteboard background color provided by an embodiment of the present invention;

[0084] Figure 7 It is a schematic diagram of the structure of a terminal device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0085] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0086] Embodiment 1

[0087] See also Figure 1 , Figure 1 FIG. 1 is a flow chart of a physical whiteboard perspective method provided by an embodiment of the present invention. Figure 1 As shown, this embodiment includes steps 101 to 104, and each step is specifically as follows:

[0088] Step 101: acquiring virtual images of multiple frames of physical whiteboard images, and performing preset color enhancement processing on the virtual whiteboard of each frame of the physical whiteboard image to obtain an enhanced image of each frame of the physical whiteboard image.

[0089] As an improvement of the above scheme, the virtual images of multiple frames of physical whiteboard images are obtained, and the virtual whiteboard of each frame of the physical whiteboard image is subjected to preset color enhancement processing to obtain an enhanced image of each frame of the physical whiteboard image, specifically: according to the virtual whiteboard of each frame of the physical whiteboard image, the background color of each frame of the physical whiteboard image is extracted to obtain the initial pixels of each point in each frame of the physical whiteboard image; according to the preset pixel activation function, the initial pixels of each point in each frame of the physical whiteboard image are respectively color enhanced.

[0090] As an improvement to the above solution, the background color of the virtual whiteboard of each frame of the physical whiteboard image can be extracted, and the pixel distribution corresponding to the virtual whiteboard can be obtained. The obtained pixel distribution can be mean filtered to obtain the initial pixel of each point.

[0091] As an improvement of the above solution, the preset pixel activation function is to ensure the purity of the whiteboard color while increasing the color contrast. The formula is as follows:

[0092]

[0093] Among them, a, b, c are adjustable coefficients, and thresh is an adaptive threshold, which is linearly related to the whiteboard background color. Considering that the whiteboard may have uneven colors due to lighting, if thresh is fixed, the dark areas of the whiteboard may become darker and the color enhancement effect cannot be achieved. Therefore, in the previous step, the whiteboard background color is pre-acquired to obtain the image threshold of each area. Different activation functions are applied to different pixels on the whiteboard to achieve the purpose of uniform color enhancement. Considering the high computational complexity of the activation function, the threshold is pre-stored and any P is calculated. ori →P new Pixel mapping is performed to facilitate subsequent color enhancement by table lookup.

[0094] Step 102: In the enhanced image of each frame of the physical whiteboard image, based on the enhanced image of the virtual whiteboard of the current frame, the enhanced image of the virtual whiteboard of the previous frame and the background image of the virtual whiteboard of the first frame, in combination with a preset weighted algorithm, a motion image of each frame of the physical whiteboard image and a color difference image of each frame of the physical whiteboard image are obtained respectively; wherein the background image of the virtual whiteboard of the first frame is extracted according to a preset algorithm to form the background image of the virtual whiteboard of the first frame of the physical whiteboard.

[0095] In this embodiment, step 102 is specifically as follows: each frame of the enhanced image is reduced to one-fourth of the original image size, and the enhanced image of the virtual whiteboard of the current frame after reduction is directly subtracted from the enhanced image of the virtual whiteboard of the previous frame after reduction, and then weighted addition is performed to obtain an absolute value. When the pixel motion change is greater than the motion threshold, the value of the corresponding position of the motion map is increased by 1, otherwise it is cleared to zero, and the formula is:

[0096]

[0097] Among them, C small (x) shows the YUV vector of the enhanced image of the virtual whiteboard of the current frame after reduction, P small (x) represents the YUV vector of the enhanced image of the previous frame of the virtual whiteboard after reduction, W YUV Indicates the weight of the YUV component, D threshrepresents the motor threshold;

[0098] The enhanced image of the virtual whiteboard in the current frame after reduction is subtracted from the background image of the virtual whiteboard in the first frame after reduction, and then weighted addition is performed to obtain the absolute value. When the chromaticity change is greater than the color difference threshold, the value of the corresponding position of the color difference map is 0, otherwise it is 1. The formula is:

[0099]

[0100] Among them, C small (x) represents the YUV vector of the reduced enhanced image of the virtual whiteboard of the current frame, and W small (x) represents the YUV vector of the background image of the first frame of the virtual whiteboard after reduction, W′ YUV Indicates the weight of the YUV component, H thresh Indicates the color difference threshold.

[0101] As an improvement of the above scheme, the background color image of the first frame of the virtual whiteboard is specifically as follows: according to the virtual whiteboard of each frame of the physical whiteboard image, the Y component of the virtual whiteboard of the first frame of the physical whiteboard image is subjected to sliding window blocking of a preset size to obtain multiple blocks; the multiple blocks are subjected to maximum heap sorting, and the mean of the Y component values ​​corresponding to the blocks whose sorting is within the preset range is extracted to obtain the background color image of the first frame of the virtual whiteboard. In order to better illustrate this improved scheme, the following example is provided for illustration: let the resolution of the virtual whiteboard be 1080p, and the Y component of the virtual whiteboard is subjected to sliding window blocking with a step size of 1, and the sliding window size is 16×16. After the sliding window, the Y component of the virtual whiteboard will be divided into 1920x1080 blocks, each block contains 16×16 pixels, and each block is subjected to maximum heap sorting, and the top 10% of the largest Y values ​​are extracted, and the average is taken, and the result after taking the average is used as the whiteboard background color; Figure 6 This is a schematic diagram of extracting the whiteboard background color. The original image is divided into blocks by sliding window, and the local maximum is calculated, and finally the average value is taken.

[0102] Step 103: Generate a foreground mask of each frame of the physical whiteboard image according to the motion map and the color difference map by using a median filter and a dilation algorithm.

[0103] In this embodiment, step 103 is specifically as follows: combining the color difference map and the motion map, and satisfying that the motion map is greater than the threshold and the color difference map is 1, the corresponding position value of the small foreground mask of 1 / 4 resolution is 1, otherwise it is 0. The small foreground mask M generated by using the median filter and dilation algorithm is small , and then use the nearest neighbor to scale to the original image size M, the formula is:

[0104]

[0105] M=Resize(dilate(M small ))

[0106] Step 104: according to the foreground mask of the current frame physical whiteboard image, the enhanced image of the current frame physical whiteboard image, and the full perspective image of the previous frame physical whiteboard image, a preset coefficient is used for weighted fusion to obtain a semi-perspective image of the current frame physical whiteboard image;

[0107] As an improvement of the above scheme, the foreground mask of the current frame physical whiteboard image, the enhanced image of the current frame physical whiteboard image, and the full perspective image of the previous frame physical whiteboard image are weighted and fused with preset coefficients to obtain a semi-perspective image of the current frame physical whiteboard image. Specifically, according to the foreground mask of the current frame physical whiteboard image, the enhanced image of the current frame physical whiteboard image, and the full perspective image of the previous frame physical whiteboard image, the image fusion is performed in combination with preset coefficients to obtain the full perspective image of the current frame physical whiteboard. The formula is:

[0108]

[0109] Among them, B(x) is the full perspective image of the physical whiteboard in the current frame, A(x) is the full perspective image of the physical whiteboard in the previous frame, C(x) is the enhanced image of the physical whiteboard in the current frame, and M(x) is the foreground mask of the physical whiteboard in the current frame. According to the full perspective image of the physical whiteboard in the current frame and the enhanced image of the physical whiteboard in the current frame, the secondary image fusion is performed in combination with the preset coefficients to obtain the semi-perspective image of the physical whiteboard in the current frame. The formula is:

[0110]

[0111] Among them, B(x) is the full perspective image of the physical whiteboard in the current frame, C(x) is the enhanced image of the physical whiteboard in the current frame, and R(x) is the semi-perspective image of the physical whiteboard in the current frame.

[0112] This embodiment not only improves the accuracy of the invention by processing the format of the image and the threshold storage processing of color enhancement, but also reduces the amount of calculation in the process, so as to improve the operating efficiency of the invention. So that users can have a better experience when using the invention. Through the virtual whiteboard generation method and the physical whiteboard perspective method, the whiteboard false detection rate is reduced while the perspective performance is improved.

[0113] See also Figure 3 , Figure 3 FIG. 1 is a flow chart of a method for generating a virtual whiteboard provided by an embodiment of the present invention. Figure 3 As shown, this embodiment includes steps 201 to 205, and each step is specifically as follows:

[0114] Step 301: Acquire a first frame of physical whiteboard image, and extract a set of edge points of the first frame of physical whiteboard image.

[0115] As an improvement of the above scheme, the method of acquiring multiple frames of physical whiteboard images and extracting the edge point set of each frame of the physical whiteboard image is as follows: after the camera collects multiple frames of original physical whiteboard images, the multiple frames of original physical whiteboard images are formatted to obtain the multiple frames of physical whiteboard images; the Y component of each frame of the physical whiteboard image is extracted as a grayscale image, and Gaussian denoising is performed, and the gradients of the x and y directions of each Y component are extracted according to the Sobel operator; the gradients of the x and y directions are weighted and thresholded, and the obtained results are binarized to obtain the edge point set of each frame of the physical whiteboard image. It should be noted that the obtained results are binarized as the edge features of the image, 0 represents not a boundary, and 1 represents a boundary.

[0116] As an improvement to the above solution, the format of the image collected by the camera is set to YUVI420 format to reduce the complexity of algorithm processing.

[0117] Step 302: According to a preset Huffman line detection algorithm, the edge point set is mapped into a first line set, and the number of intersection overlaps of each line in the first line set is calculated to obtain a valid line set of the first frame physical whiteboard image; wherein the number of intersection overlaps of the lines in the valid line set is greater than a filtering threshold.

[0118] As an improvement of the above scheme, according to the preset Huffman line detection algorithm, the edge point set is mapped into a first line set, and the number of intersection overlaps of each line in the first line set is calculated to obtain a valid line set of the first frame physical whiteboard image, specifically: constructing a Huffman space, mapping the edge point set N into multiple straight line segments N H1 、N H2 ,…,N Hn , the function expression of each straight line is: Wherein, θ∈[0°,360°], thereby obtaining the first straight line set of the first frame physical whiteboard image; according to the preset overlap resolution value, the intersection coordinates of the first straight line set are calculated, and the number of overlaps of each intersection is obtained; in the first frame physical whiteboard image, the valid intersections are mapped to valid straight lines in the Euler space, thereby obtaining the valid straight line set; wherein the valid intersection is an intersection whose number of overlaps is greater than the overlap threshold. It should be noted that the set of all straight lines passing through a certain point in space is reflected as a straight line in the Huffman space.

[0119] As an improvement of the above solution, the preset overlap resolution value may be set to 3 pixels, that is, the intersection coordinates within a range of 3×3 are considered to be the same intersection.

[0120] Step 303: combining and obtaining a whiteboard area of ​​the first frame of the physical whiteboard image according to the valid straight line set of the first frame of the physical whiteboard image;

[0121] As an improvement of the above scheme, before the combination obtains the whiteboard area of ​​the first frame of the physical whiteboard image, it also includes: according to the number of overlaps of the overlapping straight lines in the valid straight line set, through a preset filtering algorithm, filtering the straight lines with a small number of overlaps and incorrect angles in the valid straight line set to obtain the whiteboard straight line set of the first frame of the physical whiteboard image.

[0122] As an improvement of the above solution, the preset filtering algorithm can be a non-maximum suppression algorithm based on angle and straight line distance to filter the covered straight lines. The formula for straight line overlap is as follows:

[0123]

[0124] Among them, α and β are weighting coefficients; after filtering the covered straight lines, make an angle judgment to ensure the right boundary of the whiteboard: Upper Boundary And so on, where thresh l is the adaptive threshold.

[0125] As an improvement of the above scheme, the combination obtains the whiteboard area of ​​the first frame of the physical whiteboard image, specifically: according to the whiteboard straight line set, the combination obtains the valid quadrilateral set of the first frame of the physical whiteboard image; wherein the valid quadrilateral is a quadrilateral with an area greater than a preset area value; the number of overlaps of the quadrilaterals in the valid quadrilateral set is counted, and the area represented by the quadrilateral with the highest overlap number is selected as the whiteboard area of ​​the first frame of the physical whiteboard image. For better explanation: traverse the valid straight lines in the whiteboard straight line set to form (n is a valid straight line) quadrilateral M, the filtering area is lower than thresh m quadrilaterals, count the number of overlaps of all valid quadrilaterals The area with the highest number of overlaps is selected as the whiteboard area.

[0126] Step 304: Perform homography transformation processing on the whiteboard area to obtain a virtual whiteboard of the first frame of physical whiteboard image.

[0127] As an improvement of the above scheme, the whiteboard area is subjected to homography transformation processing to obtain a virtual whiteboard of the first frame of the physical whiteboard image, specifically: four corner points of the whiteboard area are calculated, and four coordinate points of the whiteboard area are obtained according to the camera model and the camera focal length; based on the four corner points of the whiteboard area and the four coordinate points of the whiteboard area, high-performance homography transformation is performed in YUV space in combination with memory optimization and assembly optimization, so as to obtain a virtual whiteboard of each frame of the physical whiteboard image.

[0128] Step 305: The virtual whiteboard of the physical whiteboard of the first frame can be used to determine the virtual whiteboard of the physical whiteboard of other frames, so as to apply the perspective method of the physical whiteboard as described in the present invention to perform whiteboard perspective on the virtual whiteboard of the physical whiteboard images of all frames; after the virtual whiteboard of the physical whiteboard of the first frame is determined, the camera position remains unchanged, and the physical whiteboard images of other frames can directly adopt the parameters of the virtual whiteboard of the physical whiteboard of the first frame to generate the virtual whiteboard of the physical whiteboard images of other frames, without the need to obtain and output the virtual whiteboard of the physical whiteboard of each frame.

[0129] See also Figure 5 , Figure 5 A schematic flow chart of a method for generating a virtual whiteboard provided in another embodiment of the present invention.

[0130] Embodiment 2

[0131] See also Figure 2 , Figure 2 It is a structural schematic diagram of a virtual whiteboard generation device provided by an embodiment of the present invention, comprising: an image enhancement module 201, a fusion module 202, a foreground mask module 203 and a perspective module 204;

[0132] The image enhancement module 201 is used to obtain virtual images of multiple frames of physical whiteboard images, and obtain an enhanced image of each frame of the physical whiteboard image after performing a preset color enhancement process on the virtual whiteboard of each frame of the physical whiteboard image;

[0133] The fusion module 202 is used to obtain, in the enhanced image of each frame of the physical whiteboard image, a motion image of each frame of the physical whiteboard image according to the enhanced image of the virtual whiteboard of the current frame and the enhanced image of the virtual whiteboard of the previous frame, in combination with a preset weighting algorithm; obtain a color difference image of each frame of the physical whiteboard image according to the enhanced image of the virtual whiteboard of the current frame and the background image of the virtual whiteboard of the first frame, in combination with a preset weighting algorithm; wherein the background image of the virtual whiteboard of the first frame is extracted according to the preset algorithm to obtain the background image of the virtual whiteboard of the physical whiteboard of the first frame;

[0134] The foreground mask module 203 is used to generate a foreground mask of each frame of the physical whiteboard image according to the motion map and the color difference map by using a median filter and a dilation algorithm;

[0135] The perspective module 204 is used to obtain a semi-perspective image of the current frame physical whiteboard image by weighted fusion using preset coefficients according to the foreground mask of the current frame physical whiteboard image, the enhanced image of the current frame physical whiteboard image, and the full perspective image of the previous frame physical whiteboard image.

[0136] As an improvement of the above scheme, the background image of the first frame of the virtual whiteboard is specifically as follows: according to the virtual whiteboard of each frame of the physical whiteboard image, the Y component of the virtual whiteboard of the first frame of the physical whiteboard image is divided into sliding windows of a preset size to obtain multiple blocks; the multiple blocks are sorted by a maximum heap, and the mean of the Y component values ​​corresponding to the blocks sorted in a preset range are extracted to obtain the background image of the first frame of the virtual whiteboard.

[0137] As an improvement of the above solution, the image enhancement module 201 includes: a first pixel unit and a second pixel unit;

[0138] The first pixel unit is used to extract the background color of each frame of the physical whiteboard image according to the virtual whiteboard of each frame of the physical whiteboard image, and obtain the initial pixel of each point in each frame of the physical whiteboard image;

[0139] The second pixel unit is used to perform color enhancement on the initial pixel of each point in each frame of the physical whiteboard image according to a preset pixel activation function.

[0140] As an improvement of the above solution, the fusion module 202 includes: a motion image unit and a color difference image unit;

[0141] The motion map unit is used to reduce each frame of the enhanced image to one-fourth of the original image size, and directly perform weighted addition on the difference between the enhanced image of the virtual whiteboard of the current frame after reduction and the enhanced image of the virtual whiteboard of the previous frame after reduction to obtain the absolute value. When the pixel motion change is greater than the motion threshold, the value of the corresponding position of the motion map is increased by 1, otherwise it is cleared to zero. The formula is:

[0142]

[0143] Among them, C small (x) shows the YUV vector of the enhanced image of the virtual whiteboard of the current frame after reduction, P small (x) represents the YUV vector of the enhanced image of the previous frame of the virtual whiteboard after reduction, W YUV Indicates the weight of the YUV component, D thresh represents the motor threshold;

[0144] The color difference map unit is used to perform weighted addition after subtracting the enhanced image of the virtual whiteboard of the current frame after reduction and the background color image of the virtual whiteboard of the first frame after reduction to obtain an absolute value. When the chromaticity change is greater than the color difference threshold, the value of the corresponding position of the color difference map is 0, otherwise it is 1. The formula is:

[0145]

[0146] Among them, C small (x) represents the YUV vector of the reduced enhanced image of the virtual whiteboard of the current frame, and W small (x) represents the YUV vector of the background image of the first frame of the virtual whiteboard after reduction, W′ YUV Indicates the weight of the YUV component, H thresh Indicates the color difference threshold.

[0147] As an improvement of the above solution, the perspective module 204 includes: a full perspective unit and a semi-perspective unit;

[0148] The full perspective unit is used to perform image fusion based on the foreground mask of the current frame physical whiteboard image, the enhanced image of the current frame physical whiteboard image, and the full perspective image of the previous frame physical whiteboard image in combination with a preset coefficient, so as to obtain the full perspective image of the current frame physical whiteboard. The formula is:

[0149]

[0150] Wherein, B(x) is the full perspective image of the physical whiteboard in the current frame, A(x) is the full perspective image of the physical whiteboard in the previous frame, C(x) is the enhanced image of the physical whiteboard in the current frame, and M(x) is the foreground mask of the physical whiteboard in the current frame.

[0151] The semi-perspective unit is used to perform secondary image fusion based on the full perspective image of the current frame physical whiteboard and the enhanced image of the current frame physical whiteboard image in combination with a preset coefficient, so as to obtain a semi-perspective image of the current frame physical whiteboard image. The formula is:

[0152]

[0153] Among them, B(x) is the full perspective image of the physical whiteboard in the current frame, C(x) is the enhanced image of the physical whiteboard in the current frame, and R(x) is the semi-perspective image of the physical whiteboard in the current frame.

[0154] See also Figure 4 , Figure 4 A physical whiteboard perspective device provided by an embodiment of the present invention comprises: an edge point extraction module 401, a straight line extraction module 402, a combination module 403, a transformation module 404 and a physical whiteboard perspective module 405;

[0155] The edge point extraction module 401 is used to obtain a first frame of physical whiteboard image and extract an edge point set of the first frame of physical whiteboard image;

[0156] The line extraction module 402 is used to map the edge point set into a first line set according to a preset Huffman line detection algorithm, and calculate the number of intersection overlaps of each line in the first line set to obtain a valid line set of the first frame of the physical whiteboard image; wherein the number of intersection overlaps of the lines in the valid line set is greater than a filtering threshold;

[0157] The combining module 403 is used to combine and obtain the whiteboard area of ​​the first frame of the physical whiteboard image according to the valid straight line set of the first frame of the physical whiteboard image;

[0158] The transformation module 404 is used to perform homography transformation processing on the whiteboard area to obtain a virtual whiteboard of the first frame of physical whiteboard image;

[0159] The physical whiteboard perspective module 405 is used to determine the virtual whiteboards of the physical whiteboards of other frames based on the virtual whiteboard of the physical whiteboard of the first frame, so as to apply the physical whiteboard perspective method of the present invention to perform whiteboard perspective on the virtual whiteboards of the physical whiteboard images of all frames.

[0160] As an improvement of the above solution, the straight line extraction module 402 includes: a first mapping unit, an overlapping unit and a second mapping unit;

[0161] The first mapping unit is used to construct a Huffman space to map the edge point set N into a plurality of straight lines N H1 、N H2 ,…,N Hn , the function expression of each straight line is: Wherein, θ∈[0°,360°], thereby obtaining a first straight line set of the first frame of the physical whiteboard image;

[0162] The overlapping unit is used to calculate the coordinates of the intersection points of the first set of straight lines according to a preset overlapping resolution value, and obtain the number of overlapping times of each intersection point;

[0163] The second mapping unit is used to map valid intersections in the first frame of the physical whiteboard image into valid straight lines in the Euler space, so as to obtain the valid straight line set; wherein the valid intersections are intersections whose overlap times are greater than an overlap threshold.

[0164] As an improvement of the above scheme, before the combination obtains the whiteboard area of ​​the first frame of the physical whiteboard image, it also includes: according to the number of overlaps of the overlapping straight lines in the valid straight line set, through a preset filtering algorithm, filtering the straight lines with a small number of overlaps and incorrect angles in the valid straight line set to obtain the whiteboard straight line set of the first frame of the physical whiteboard image.

[0165] As an improvement of the above solution, the combination module 403 includes: a first combination unit and a second combination unit;

[0166] The first combination unit is used to combine the whiteboard straight line set to obtain a valid quadrilateral set of the first frame of the physical whiteboard image; wherein the valid quadrilateral is a quadrilateral whose area is greater than a preset area value;

[0167] The first combining unit is used to count the overlapping times of the quadrilaterals in the valid quadrilateral set, and select the area represented by the quadrilateral with the highest overlapping times as the whiteboard area of ​​the first frame of the physical whiteboard image.

[0168] As an improvement of the above solution, the transformation module 404 includes: a first processing unit and a second processing unit;

[0169] The first processing unit is used to calculate four corner points of the whiteboard area, and obtain four coordinate points of the whiteboard area according to the camera model and the camera focal length;

[0170] The second processing unit is used to perform high-performance homography transformation in YUV space according to the four corner points of the whiteboard area and the four coordinate points of the whiteboard area, combined with memory optimization and assembly optimization, so as to obtain a virtual whiteboard of each frame of the physical whiteboard image.

[0171] After the virtual whiteboard generation device in this embodiment extracts the edge point set through the edge point extraction module, it is input into the straight line extraction module for straight line mapping, and then all the straight lines are combined into a whiteboard area through the combination module, and finally converted into a virtual whiteboard through the transformation module, thereby improving the accuracy of extracting the virtual whiteboard. After the physical whiteboard perspective device obtains the virtual whiteboard of all physical whiteboard images through the virtual whiteboard extraction module, it uses the image enhancement module to enhance all virtual whiteboards, and then uses the fusion module to fuse the background image extracted from the first frame of the physical whiteboard image, the virtual whiteboard of the current frame, and the virtual whiteboard of the previous frame to calculate the color difference map and motion map, and generates a foreground mask by combining the color difference map and the motion map through the foreground mask module, and finally realizes the smooth perspective effect of the whiteboard through the perspective module. This embodiment combines the virtual whiteboard generation device and the physical whiteboard perspective device to reduce the false detection rate of whiteboard detection and realize a beautiful perspective function.

[0172] Embodiment 3

[0173] See also Figure 7 , Figure 7 It is a schematic diagram of the structure of a terminal device provided in one embodiment of the present invention.

[0174] A terminal device of this embodiment includes: a processor 701, a memory 702, and a computer program stored in the memory 702 and executable on the processor 701. When the processor 701 executes the computer program, the steps of the above-mentioned virtual whiteboard generation method and physical whiteboard perspective method in the embodiment are implemented, for example Figure 1 Alternatively, when the processor executes the computer program, the functions of each module in the above-mentioned device embodiments are implemented, for example: Figure 2 All modules of the perspective device of the physical whiteboard are shown.

[0175] In addition, an embodiment of the present invention further provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the virtual whiteboard generation method and the physical whiteboard perspective method as described in any of the above embodiments.

[0176] Those skilled in the art will understand that the schematic diagram is merely an example of a terminal device and does not constitute a limitation on the terminal device. The terminal device may include more or fewer components than shown in the diagram, or a combination of certain components, or different components. For example, the terminal device may also include input and output devices, network access devices, buses, etc.

[0177] The processor 701 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor 701 is the control center of the terminal device, and uses various interfaces and lines to connect various parts of the entire terminal device.

[0178] The memory 702 can be used to store the computer program and / or module, and the processor 701 implements various functions of the terminal device by running or executing the computer program and / or module stored in the memory, and calling the data stored in the memory 702. The memory 702 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0179] Wherein, if the module / unit integrated in the terminal device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0180] It should be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the accompanying drawings of the device embodiments provided by the present invention, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines. A person of ordinary skill in the art may understand and implement it without paying any creative effort.

[0181] The above is a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the principle of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A perspective method for physical whiteboards, It is characterized in that include: Acquire virtual images of multiple frames of physical whiteboard images, and perform preset color enhancement processing on the virtual whiteboard of each frame of the physical whiteboard image to obtain an enhanced image of each frame of the physical whiteboard image; In the enhanced image of each frame of the physical whiteboard image, a motion image of each frame of the physical whiteboard image is obtained according to the enhanced image of the virtual whiteboard of the current frame and the enhanced image of the virtual whiteboard of the previous frame, in combination with a preset weighting algorithm; a color difference image of each frame of the physical whiteboard image is obtained according to the enhanced image of the virtual whiteboard of the current frame and the background image of the virtual whiteboard of the first frame, in combination with a preset weighting algorithm; wherein the background image of the virtual whiteboard of the first frame is extracted according to a preset algorithm to obtain the background image of the virtual whiteboard of the physical whiteboard of the first frame; Generate a foreground mask of each frame of the physical whiteboard image by using a median filter and a dilation algorithm according to the motion map and the color difference map; According to the foreground mask of the current frame physical whiteboard image, the enhanced image of the current frame physical whiteboard image, and the full perspective image of the previous frame physical whiteboard image, a preset coefficient is used for weighted fusion to obtain the full perspective image of the current frame physical whiteboard; the full perspective image of the current frame physical whiteboard and the enhanced image of the current frame physical whiteboard image are combined with preset coefficients for secondary image fusion to obtain a semi-perspective image of the current frame physical whiteboard image.

2. The physical whiteboard perspective method according to claim 1, It is characterized in that The background color image of the first frame of the virtual whiteboard is specifically: According to the virtual whiteboard of each frame of the physical whiteboard image, the Y component of the virtual whiteboard of the first frame of the physical whiteboard image is divided into sliding windows of a preset size to obtain a plurality of blocks; The plurality of blocks are sorted by a maximum heap sort, and the average of the Y component values ​​corresponding to the blocks whose sorting is within a preset range is extracted to obtain the background color image of the first frame of the virtual whiteboard.

3. The physical whiteboard perspective method according to claim 2, It is characterized in that The step of acquiring virtual images of multiple frames of physical whiteboard images and performing a preset color enhancement process on the virtual whiteboard of each frame of the physical whiteboard image to obtain an enhanced image of each frame of the physical whiteboard image is specifically as follows: Extracting the background color of each frame of the physical whiteboard image according to the virtual whiteboard of each frame of the physical whiteboard image, and obtaining the initial pixel of each point in each frame of the physical whiteboard image; According to a preset pixel activation function, color enhancement is performed on the initial pixel of each point in each frame of the physical whiteboard image.

4. The physical whiteboard perspective method according to claim 1, It is characterized in that In the enhanced image of each frame of the physical whiteboard image, according to the enhanced image of the current frame of the virtual whiteboard, the enhanced image of the previous frame of the virtual whiteboard and the background image of the first frame of the virtual whiteboard, in combination with a preset weighting algorithm, a motion image of each frame of the physical whiteboard image and a color difference image of each frame of the physical whiteboard image are obtained respectively, specifically: Each frame of the enhanced image is reduced to one-fourth of the original image size. The enhanced image of the virtual whiteboard of the current frame after reduction is directly subtracted from the enhanced image of the virtual whiteboard of the previous frame after reduction, and then weighted addition is performed to obtain the absolute value. When the pixel motion change is greater than the motion threshold, the value of the corresponding position of the motion map is increased by 1, otherwise it is cleared to zero. The formula is: Among them, C small (x) shows the YUV vector of the enhanced image of the virtual whiteboard of the current frame after reduction, P small (x) represents the YUV vector of the enhanced image of the previous frame of the virtual whiteboard after reduction, W YUV Denotes the weight of the motion image YUV component, D thresh represents the motor threshold; The enhanced image of the virtual whiteboard in the current frame after reduction is subtracted from the background image of the virtual whiteboard in the first frame after reduction, and then weighted addition is performed to obtain the absolute value. When the chromaticity change is greater than the color difference threshold, the value of the corresponding position of the color difference map is 0, otherwise it is 1. The formula is: Among them, C small (x) represents the YUV vector of the reduced enhanced image of the virtual whiteboard of the current frame, and W small (x) represents the YUV vector of the background image of the first frame of the virtual whiteboard after reduction, W′ YUV Indicates the weight of the YUV component of the color difference map, H thresh Indicates the color difference threshold.

5. The physical whiteboard perspective method according to claim 1, It is characterized in that The method is to obtain a semi-perspective image of the current frame of the physical whiteboard image by weighted fusion using a preset coefficient according to the foreground mask of the current frame of the physical whiteboard image, the enhanced image of the current frame of the physical whiteboard image, and the full perspective image of the previous frame of the physical whiteboard image, specifically: According to the foreground mask of the current frame physical whiteboard image, the enhanced image of the current frame physical whiteboard image, and the full perspective image of the previous frame physical whiteboard image, image fusion is performed in combination with the preset coefficients to obtain the full perspective image of the current frame physical whiteboard. The formula is: Wherein, B(x) is the full perspective image of the physical whiteboard in the current frame, A(x) is the full perspective image of the physical whiteboard in the previous frame, C(x) is the enhanced image of the physical whiteboard in the current frame, and M(x) is the foreground mask of the physical whiteboard in the current frame. According to the full perspective image of the current frame physical whiteboard and the enhanced image of the current frame physical whiteboard image, a secondary image fusion is performed in combination with a preset coefficient to obtain a semi-perspective image of the current frame physical whiteboard image. The formula is: Among them, B(x) is the full perspective image of the physical whiteboard in the current frame, C(x) is the enhanced image of the physical whiteboard in the current frame, and R(x) is the semi-perspective image of the physical whiteboard in the current frame.

6. A method for generating a virtual whiteboard, It is characterized in that include: Acquire a first frame of physical whiteboard image, and extract a set of edge points of the first frame of physical whiteboard image; According to a preset Huffman line detection algorithm, the edge point set is mapped into a first line set, and the number of intersection overlaps of each line in the first line set is calculated to obtain a valid line set of the first frame of the physical whiteboard image; wherein the number of intersection overlaps of the lines in the valid line set is greater than a filtering threshold; According to the valid straight line set of the first frame of the physical whiteboard image, a whiteboard area of ​​the first frame of the physical whiteboard image is obtained by combination; Performing homography transformation processing on the whiteboard area to obtain a virtual whiteboard of the first frame of physical whiteboard image; The virtual whiteboard of the physical whiteboard of the first frame can be used to determine the virtual whiteboard of the physical whiteboard of other frames, thereby applying the physical whiteboard perspective method as described in any one of claims 1 to 5 to perform whiteboard perspective on the virtual whiteboard of the physical whiteboard images of all frames.

7. The method for generating a virtual whiteboard according to claim 6, It is characterized in that According to the preset Huffman line detection algorithm, the edge point set is mapped into a first line set, and the number of intersection overlaps of each line in the first line set is calculated to obtain a valid line set of the first frame physical whiteboard image, specifically: Construct a Huffman space and map the edge point set N into multiple straight lines N H1 、N H2 ,…,N Hn , the function expression of each straight line is: Wherein, θ∈[0°,360°], thereby obtaining a first straight line set of the first frame of the physical whiteboard image; According to a preset overlap resolution value, the coordinates of the intersections of the first set of straight lines are calculated, and the number of overlaps of each intersection is obtained; In the first frame of the physical whiteboard image, valid intersection points are mapped to valid straight lines in the Euler space, thereby obtaining the valid straight line set; wherein the valid intersection points are intersection points whose overlap times are greater than an overlap threshold.

8. The method for generating a virtual whiteboard according to claim 6, It is characterized in that Before the combination obtains the whiteboard area of ​​the first frame of the physical whiteboard image, it also includes: according to the number of overlaps of the overlapping straight lines in the valid straight line set, using a preset filtering algorithm, filtering the straight lines with a small number of overlaps and incorrect angles in the valid straight line set to obtain the whiteboard straight line set of the first frame of the physical whiteboard image.

9. The method for generating a virtual whiteboard according to claim 8, It is characterized in that The combination obtains the whiteboard area of ​​the first frame of the physical whiteboard image, specifically: According to the whiteboard straight line set, a valid quadrilateral set of the first frame of the physical whiteboard image is obtained by combination; wherein the valid quadrilateral is a quadrilateral whose area is greater than a preset area value; The number of overlapping times of quadrilaterals in the valid quadrilateral set is counted, and the area represented by the quadrilateral with the highest overlapping time is selected as the whiteboard area of ​​the first frame of the physical whiteboard image.

10. The method for generating a virtual whiteboard according to claim 6, It is characterized in that The performing homography transformation on the whiteboard area to obtain a virtual whiteboard of the first frame of the physical whiteboard image is specifically: Calculate four corner points of the whiteboard area, and obtain four coordinate points of the whiteboard area according to the camera model and the camera focal length; According to the four corner points of the whiteboard area and the four coordinate points of the whiteboard area, combined with memory optimization and assembly optimization, a high-performance homography transformation is performed in the YUV space to obtain a virtual whiteboard of each frame of the physical whiteboard image.

Images