Paper screen interaction method and device based on image, storage medium and terminal

Abstract

The invention discloses an image-based paper screen interaction method and device, a storage medium and a terminal, relates to the technical field of computer vision, and mainly aims to solve the problem that existing paper screen interaction is relatively poor in interaction timeliness. The method mainly comprises the steps that under the condition that a display screen displays interactive display data corresponding to a paper medium, an image of an area where the paper medium is located is collected, and target detection is conducted on the image; under the condition that the target detection result contains the hand image, extracting hand contour information and finger curvature information from the hand image; under the condition that the finger bending degree information meets a preset page turning gesture condition, according to hand contour information corresponding to the multiple frames of hand images, a gesture moving track is calculated, and action recognition is conducted according to the gesture moving track; and when the action recognition result is the page turning action, displaying the synchronous page turning animation in the display screen according to the page turning direction in the action recognition result. The method is mainly used for identifying page turning actions and interacting.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to an image-based paper-screen interaction method and device, a storage medium, and a terminal. Background Art

[0002] With the rapid development of digital technology, the demand for interaction between paper media and digital screens is increasing. Traditional paper has the advantages of being convenient, easy to write and read, while digital devices provide powerful computing power and rich multimedia content. Therefore, combining the two to achieve seamless paper-screen interaction can greatly enhance the user experience.

[0003] At present, the existing paper-screen interaction mainly recognizes the image of the paper content and displays the corresponding content on the screen according to the image recognition result. However, this method requires the user to complete the page turning action before performing image analysis and recognition, which makes the display on the screen have a certain time lag, resulting in poor timeliness of paper-screen interaction. Summary of the invention

[0004] In view of this, the present invention provides an image-based paper-screen interaction method and device, a storage medium, and a terminal, the main purpose of which is to solve the problem of poor timeliness and interaction of existing paper-screen interactions.

[0005] According to one aspect of the present invention, there is provided an image-based paper-screen interaction method, comprising:

[0006] When the display screen displays interactive display data corresponding to the paper medium, an image of the area where the paper medium is located is collected, and target detection is performed on the image;

[0007] When the target detection result includes a hand image, extracting hand contour information and finger curvature information from the hand image;

[0008] When the finger curvature information satisfies a preset page-turning gesture condition, the gesture movement trajectory is calculated according to the hand contour information corresponding to the multiple frames of the hand images, and the action recognition is performed according to the gesture movement trajectory;

[0009] When the action recognition result is a page turning action, a synchronous page turning animation and interactive display data of the unfolded page after turning the page are displayed on the display screen according to the page turning direction in the action recognition result.

[0010] Furthermore, the extracting hand contour information and finger curvature information from the hand image includes:

[0011] Performing hand contour detection on the hand image to obtain hand contour information, wherein the hand contour information includes palm coordinate information, the number of fingers and key point coordinate information of each finger, and the key point coordinate information includes at least two of fingertip coordinates, fingertip joint coordinates, interphalangeal joint coordinates, and metacarpophalangeal joint coordinates;

[0012] For each finger, calculate the coordinate information of at least one bending identification segment according to the key point coordinate information, wherein the bending identification segment includes a first identification segment between the fingertip and the fingertip joint, a second identification segment between the fingertip joint and the interphalangeal joint, and a third identification segment between the interphalangeal joint and the metacarpophalangeal joint;

[0013] The finger curvature of each finger is calculated according to the coordinate information of at least one bending identification segment of the finger and the palm coordinate information to obtain the finger curvature information.

[0014] Furthermore, the finger curvature information includes the curvature of at least one curvature identification segment corresponding to each of the different fingers, and each of the different curvature identification segments corresponds to a different preset curvature threshold;

[0015] After extracting the hand contour information and the finger curvature information from the hand image, the method further includes:

[0016] Identify the bending state of the fingers one by one, and if the curvature of at least one bending identification segment of the finger is greater than the corresponding preset curvature threshold, determine that the finger is a bent finger;

[0017] When the number of bent fingers is greater than a preset number threshold, it is determined that the finger bending degree meets the preset page turning gesture condition.

[0018] Furthermore, the performing action recognition according to the gesture movement trajectory includes:

[0019] When the horizontal movement distance is greater than a preset horizontal movement threshold and the movement speed is greater than a preset movement speed threshold, the action recognition result is determined to be a page turning action, wherein the gesture movement trajectory includes the horizontal movement distance and the movement speed; or,

[0020] When the horizontal movement distance is greater than the preset horizontal movement threshold, if the vertical movement distance is less than the preset vertical movement threshold, and the movement speed is greater than the preset movement speed threshold, the action recognition result is determined to be a page turning action, wherein the gesture movement trajectory includes the horizontal movement distance, the vertical movement distance and the movement speed.

[0021] Furthermore, after determining that the action recognition result is a page turning action, the method further includes:

[0022] Obtaining the layout content of the unfolded page of the paper medium;

[0023] When the layout content matches the interactive display data displayed on the display screen, the interactive display data displayed on the display screen is kept unchanged.

[0024] Furthermore, the method further comprises:

[0025] When the action recognition result is a page turning action, determining the direction of the horizontal moving distance;

[0026] The page turning direction is determined according to the direction of the horizontal moving distance.

[0027] Furthermore, the performing hand contour detection on the hand image to obtain hand contour information includes:

[0028] Performing blur denoising and smoothing filtering processing on the hand image to obtain a preprocessed hand image;

[0029] Converting the preprocessed hand image to an HSV color space, performing threshold segmentation on the preprocessed hand image according to a preset skin color range in the HSV color space, and obtaining a hand skin region image;

[0030] Performing morphological operation denoising on the hand skin area image to obtain a denoised hand skin area image;

[0031] The finger contour and palm contour target detection is performed based on the denoised hand skin area image to obtain hand contour information.

[0032] Furthermore, the calculating of the gesture movement trajectory according to the finger contour information corresponding to the multiple frames of the hand images includes:

[0033] The hand image whose finger curvature satisfies the preset page-turning gesture condition is taken as the starting frame image, and the hand position difference coordinates of each adjacent frame image are calculated according to the finger contour information of each adjacent frame image after the starting frame image, and a time window is created, wherein the hand position difference coordinates are used to characterize whether the hand is moving;

[0034] If within the time window, the hand position difference coordinates represent that the hand movement has stopped, then the gesture movement trajectory is calculated based on all the hand position difference coordinates between the start frame image and the end frame image;

[0035] If the hand position difference coordinates do not indicate that the hand movement has stopped at the end of the time window, the action recognition result is determined to be a non-page turning action, and the calculation of the hand position difference coordinates of adjacent frame images is stopped.

[0036] According to another aspect of the present invention, there is provided an image-based paper-screen interactive device, comprising:

[0037] A detection module, used to collect an image of the area where the paper medium is located and perform target detection on the image when the display screen displays interactive display data corresponding to the paper medium;

[0038] An extraction module, used for extracting hand contour information and finger curvature information from the hand image when the target detection result includes the hand image;

[0039] A recognition module, configured to calculate a gesture movement trajectory based on hand contour information corresponding to a plurality of frames of hand images when the finger curvature information satisfies a preset page-turning gesture condition, and perform action recognition based on the gesture movement trajectory;

[0040] The interactive display module is used to display a synchronous page turning animation and interactive display data of the unfolded page after turning the page on the display screen according to the page turning direction in the action recognition result when the action recognition result is a page turning action.

[0041] Furthermore, the extraction module comprises:

[0042] a detection unit, configured to perform hand contour detection on the hand image to obtain hand contour information, wherein the hand contour information includes palm coordinate information, the number of fingers and key point coordinate information of each finger, and the key point coordinate information includes at least two of fingertip coordinates, fingertip joint coordinates, interphalangeal joint coordinates, and metacarpophalangeal joint coordinates;

[0043] a first calculation unit, for calculating, for each finger, coordinate information of at least one bending identification segment according to the key point coordinate information, wherein the bending identification segment includes a first identification segment between a fingertip and a fingertip joint, a second identification segment between a fingertip joint and an interphalangeal joint, and a third identification segment between an interphalangeal joint and a metacarpophalangeal joint;

[0044] The second calculation unit is used to calculate the finger curvature of each finger according to the coordinate information of at least one bending identification segment of the finger and the palm coordinate information to obtain the finger curvature information.

[0045] Furthermore, the device also includes:

[0046] The identification module is further used to identify the bending state of the fingers one by one, and if the curvature of at least one bending identification segment of the finger is greater than the corresponding preset curvature threshold, the finger is determined to be a bent finger;

[0047] The determination module is used to determine that the finger bending degree meets the preset page turning gesture condition when the number of the bent fingers is greater than a preset number threshold.

[0048] Furthermore, the identification module includes:

[0049] A first determining unit is used to determine that the action recognition result is a page turning action when the horizontal movement distance is greater than a preset horizontal movement threshold and the movement speed is greater than a preset movement speed threshold, wherein the gesture movement trajectory includes the horizontal movement distance and the movement speed; or

[0050] The second determination unit is used to determine that the action recognition result is a page turning action when the horizontal movement distance is greater than a preset horizontal movement threshold, if the vertical movement distance is less than the preset vertical movement threshold, and the movement speed is greater than the preset movement speed threshold, wherein the gesture movement trajectory includes the horizontal movement distance, the vertical movement distance and the movement speed.

[0051] Furthermore, the device also includes:

[0052] An acquisition module, used to acquire the layout content of the unfolded page of the paper medium;

[0053] The interactive maintaining module is used to maintain the interactive display data displayed on the display screen unchanged when the layout content matches the interactive display data displayed on the display screen.

[0054] Furthermore, the device also includes:

[0055] A first determination module, used for determining the direction of the horizontal moving distance when the action recognition result is a page turning action;

[0056] The second determining module is used to determine the page turning direction according to the direction of the horizontal moving distance.

[0057] Furthermore, in a specific application scenario, the detection unit is specifically used to perform blur denoising and smoothing filtering on the hand image to obtain a preprocessed hand image;

[0058] Converting the preprocessed hand image to an HSV color space, performing threshold segmentation on the preprocessed hand image according to a preset skin color range in the HSV color space, and obtaining a hand skin region image;

[0059] Performing morphological operation denoising on the hand skin area image to obtain a denoised hand skin area image;

[0060] The finger contour and palm contour target detection is performed based on the denoised hand skin area image to obtain hand contour information.

[0061] Furthermore, the identification module includes:

[0062] A third calculation unit is used to use the hand image whose finger curvature satisfies the preset page turning gesture condition as the starting frame image, calculate the hand position difference coordinates of each adjacent frame image according to the finger contour information of each adjacent frame image after the starting frame image, and create a time window, wherein the hand position difference coordinates are used to indicate whether the hand is moving;

[0063] a fourth calculation unit, configured to calculate a gesture movement trajectory according to all hand position difference coordinates between the start frame image and the end frame image if the hand position difference coordinates indicate that the hand movement has stopped within the time window;

[0064] The fifth determination unit is used to determine that the action recognition result is a non-page turning action and stop calculating the hand position difference coordinates of adjacent frame images if the hand position difference coordinates do not represent the cessation of hand movement at the end of the time window.

[0065] According to another aspect of the present invention, a storage medium is provided, wherein at least one executable instruction is stored in the storage medium, and the executable instruction enables a processor to perform operations corresponding to the above-mentioned image-based paper-screen interaction method.

[0066] According to another aspect of the present invention, there is provided a terminal, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus;

[0067] The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the above-mentioned image-based paper-screen interaction method.

[0068] By means of the above technical solution, the technical solution provided by the embodiment of the present invention has at least the following advantages:

[0069] The present invention provides an image-based paper-screen interaction method and device, storage medium, and terminal. The embodiment of the present invention collects images of the area where the paper medium is located and performs target detection on the image when the interactive display data corresponding to the paper medium is displayed on the display screen; when the target detection result includes a hand image, hand contour information and finger curvature information are extracted from the hand image; when the finger curvature information meets the preset page-turning gesture conditions, the gesture movement trajectory is calculated based on the hand contour information corresponding to multiple frames of hand images, and action recognition is performed based on the gesture movement trajectory; when the action recognition result is a page-turning action, a synchronous page-turning animation is displayed on the display screen according to the page-turning direction in the action recognition result, and the interactive display data of the unfolded page after the page is turned. The recognition starts from the time the page-turning action is generated, which greatly advances the response time of the page-turning recognition, and at the same time, ensures the accuracy of the page-turning action recognition, thereby effectively improving the timeliness of the paper-screen interaction.

[0070] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0072] Figure 1 A flow chart of an image-based paper-screen interaction method provided by an embodiment of the present invention is shown;

[0073] Figure 2 A flowchart of another image-based paper-screen interaction method provided by an embodiment of the present invention is shown;

[0074] Figure 3 A block diagram of a paper-screen interactive device based on an image provided by an embodiment of the present invention is shown;

[0075] Figure 4 A schematic diagram of the structure of a terminal provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0076] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0077] In order to solve the problem that the paper screen interaction is less timely, the embodiment of the present invention provides an image-based paper screen interaction method, such as Figure 1 As shown, the method includes:

[0078] 101. When a display screen displays interactive display data corresponding to a paper medium, an image of an area where the paper medium is located is collected, and target detection is performed on the image.

[0079] In the embodiment of the present invention, the page turning action of the user is mainly identified. When the display screen is displaying the interactive display data corresponding to the unfolded page of the paper medium, in order to synchronize the user's page turning action on the display screen, the image of the area where the paper medium is located is captured in real time by a camera, so as to identify the user's action on the paper medium based on the user's hand action in the image, so as to synchronously display the corresponding operation on the display screen to realize paper-screen interaction. Among them, the paper medium is a paper print that needs to be interactive with the paper screen, such as a book, an exercise book, a test paper, etc. The display screen can be a display screen of a mobile display device, such as a tablet computer, a mobile phone, a learning machine, etc. The camera can be a camera device integrated in the device where the display screen is located, or it can be a camera device that has a data communication relationship with the device where the display screen is located. The embodiment of the present invention does not specifically limit the specific content of the paper medium, the display screen and the camera. The image is a video stream. After the video stream is obtained, target detection is performed on each frame of the image to determine whether the current frame image contains a hand image.

[0080] 102. When the target detection result includes a hand image, extract hand contour information and finger curvature information from the hand image.

[0081] In an embodiment of the present invention, if the target detection result does not include a hand image, it indicates that the user has not touched the paper medium with his hand and will not make any action on the paper medium. Such an image can be discarded. If the target detection result includes a hand image, it indicates that the user may make an action on the paper medium and further identification of the action intention is required. In order to identify the user's action on the paper medium, hand contour information, such as palm coordinate information, finger coordinate information, etc., is extracted, as well as curvature information of each finger, such as the curvature angle of the finger segment between the proximal segment (first phalanx) and the middle segment (second phalanx) of the finger relative to the palm plane.

[0082] 103. When the finger curvature information satisfies a preset page-turning gesture condition, calculate a gesture movement trajectory according to hand contour information corresponding to a plurality of frames of hand images, and perform action recognition according to the gesture movement trajectory.

[0083] In an embodiment of the present invention, the recognition of the user's page-turning action is decomposed into two steps: page-turning gesture recognition and page-turning action recognition. That is, the page-turning gesture recognition is performed first. When the user's action meets the preset page-turning gesture condition, the image frame corresponding to the current hand image is used as the starting frame for page-turning action recognition. The gesture movement trajectory is extracted based on the coordinate changes of the same hand position in the multiple frames of images after this starting frame, and then the user action is identified as page turning based on the movement trajectory. Among them, the preset page-turning gesture condition is determined based on the finger posture characteristics of the user when turning the page. For example, if the number of bent fingers is greater than or equal to 3, it is determined that the page-turning gesture is satisfied. The multiple frames of images can be continuous frame images (such as 1, 2, 3...) or non-continuous frame images (such as 1, 3, 5...), which are not specifically limited in the embodiment of the present invention.

[0084] It should be noted that when a user initiates a page turning action, they usually pinch a page of paper first, and then move their hand to turn the pinched page. By decomposing the recognition of the user's page turning action into page turning gesture recognition and page turning action recognition, the two significant features of page turning can be recognized separately, thereby accurately capturing the user's page turning action.

[0085] 104. When the action recognition result is a page turning action, a synchronous page turning animation and interactive display data of the unfolded page after page turning are displayed on the display screen according to the page turning direction in the action recognition result.

[0086] In the embodiment of the present invention, the action recognition result includes a page turning action and a non-page turning action. When the action recognition result is a page turning action, according to the direction of the page turning action (forward page turning or backward page turning), a page turning animation is synchronously displayed on the display screen, and the interactive display data of the page opened after the page turning is displayed on the display screen.

[0087] It should be noted that compared with the existing method of interactive page turning on the display screen based on the change of the unfolded page layout, page turning based on user gestures and actions can greatly improve the response speed to the user's page turning action, making the degree of interactive simulation closer to reality and achieving true synchronization of page turning. Especially in the application of teaching scenarios, timely page turning response can bring more full participation and ritual sense to students, thereby improving the quality of user experience.

[0088] In an embodiment of the present invention, for further explanation and limitation, as Figure 2As shown, the step of extracting hand contour information and finger curvature information from the hand image includes:

[0089] 201. Perform hand contour detection on the hand image to obtain hand contour information.

[0090] 202. For each finger, calculate coordinate information of at least one bending recognition segment according to the key point coordinate information.

[0091] 203. Calculate the finger curvature of each finger according to the coordinate information of at least one bending identification segment of the finger and the palm coordinate information to obtain finger curvature information.

[0092] In an embodiment of the present invention, the hand contour information includes palm coordinate information, the number of fingers and key point coordinate information of each finger. The key point coordinate information includes at least two of the fingertip coordinates, fingertip joint coordinates, interphalangeal joint coordinates, and metacarpophalangeal root joint coordinates. Among them, the line between any two joints in the fingertip coordinates, fingertip joint coordinates, interphalangeal joint coordinates, and metacarpophalangeal root joint coordinates can determine a segment of the finger. Using the cosine theorem, the angle between this segment of the finger and the plane where the palm is located can be calculated, and this angle is determined as the curvature of the corresponding finger. Taking the finger segment between the metacarpophalangeal root joint and the interphalangeal joint as the bending identification segment as an example, there is an angle between the bending identification segment and the palm plane. A vertical line is drawn from the interphalangeal joint point to the palm plane, which can form a right triangle with the bending identification segment as the hypotenuse. According to the cosine theorem, the angle between the bending identification segment and the palm plane can be calculated, and this angle is used as the curvature of the corresponding finger. Of course, the angles between two or more bending recognition segments and the plane where the palm is located can also be calculated, and the largest angle can be determined as the curvature of the finger to avoid the situation where the curvature of the finger cannot be recognized based on only one bending recognition segment. The finger curvature information includes the curvature of each recognized finger.

[0093] It should be noted that based on the common gestures when turning pages (five fingers touching the page, three fingers curled up, and two fingers touching the page), the first identification segment between the fingertips and fingertip joints, the second identification segment between the fingertip joints and the interphalangeal joints, and the third identification segment between the interphalangeal joints and the metacarpophalangeal joints are determined as the preferred bending identification segments. By identifying the curvature of the fingers, it is a process of capturing gestures based on the characteristics of a person's page-turning action, which can accurately identify the user's intention to turn the page, so as to provide accurate data basis for the subsequent tracking of the trajectory of the page-turning action and the determination of whether the page-turning action is completed.

[0094] In one embodiment of the present invention, for further explanation and limitation, after extracting the hand contour information and the finger curvature information from the hand image in step 1, the method further includes:

[0095] Identify the bending state of the fingers one by one, and if the curvature of at least one bending identification segment of the finger is greater than the corresponding preset curvature threshold, determine that the finger is a bent finger;

[0096] When the number of bent fingers is greater than a preset number threshold, it is determined that the finger bending degree meets the preset page turning gesture condition.

[0097] In an embodiment of the present invention, the finger curvature information includes the curvature of at least one curvature identification segment corresponding to each of the different fingers. Different curvature identification segments correspond to different preset curvature thresholds. Among them, the preset curvature threshold is configured according to the actual curvature characteristics of the finger. For example, the preset curvature threshold corresponding to the first identification segment between the fingertip and the fingertip joint can be 30°, the preset curvature threshold corresponding to the second identification segment between the fingertip joint and the interphalangeal joint can be 25°, and the preset curvature threshold corresponding to the third identification segment between the interphalangeal joint and the metacarpophalangeal joint can be 15°. After determining whether each finger is in a bent state, it is further determined whether the current gesture meets the preset page turning gesture condition according to the number of fingers in the bent state. The preset number threshold can be 3 or 2, which is specifically determined according to the curvature identification segment based on which the curvature calculation is based. For example, when the bending identification segment selects the first identification segment and the second identification segment, the corresponding preset number threshold can be 2, and when the bending identification segment selects the third identification segment, the corresponding preset number threshold can be 3.

[0098] By distinguishing the bending state of fingers according to the preset bending degree thresholds corresponding to different bending recognition segments and recognizing the page-turning gesture according to the number of bent fingers, the finger posture characteristics of the page-turning gesture can be comprehensively recognized, thereby improving the recognition accuracy of the page-turning gesture.

[0099] In one embodiment of the present invention, for further explanation and limitation, the step of performing action recognition according to the gesture movement trajectory includes:

[0100] When the horizontal movement distance is greater than a preset horizontal movement threshold and the movement speed is greater than a preset movement speed threshold, determining that the action recognition result is a page turning action; or,

[0101] When the horizontal movement distance is greater than the preset horizontal movement threshold, if the vertical movement distance is less than the preset vertical movement threshold, and the movement speed is greater than the preset movement speed threshold, the action recognition result is determined to be a page turning action.

[0102] In an embodiment of the present invention, two scenarios for page turning action recognition are provided. One is that the gesture movement trajectory includes horizontal movement distance, vertical movement distance and movement speed. That is, the horizontal and vertical movement distances of the gesture movement trajectory are judged by a preset horizontal movement threshold and a preset vertical movement threshold, and then the page turning action is determined in combination with the movement speed. Among them, the vertical movement distance can be the vertical distance difference between the starting position and the ending position in the gesture movement trajectory, or it can be the difference between the highest position and the lowest position in the gesture movement trajectory, that is, the maximum vertical distance. The page turning action is recognized by the vertical and horizontal recognition conditions and the speed recognition conditions, which can ensure the accuracy of the recognition and effectively avoid the misrecognition of non-page turning actions. This recognition method is more suitable for paper media with a smaller width, such as books, in which the maximum vertical distance of the hand movement is small during the turning process, and the hand contour will not exceed the image acquisition range during the entire page turning action.

[0103] The other is that the gesture movement trajectory includes a horizontal movement distance and a movement speed, and the page turning action is judged according to a preset horizontal movement threshold and a preset movement speed threshold. In the scenario where the paper width of the paper medium is large, such as a test paper, a wide exercise book, etc., the vertical movement distance of the page turning gesture is large, and there is a possibility that the hand contour exceeds the image acquisition range. If the preset vertical movement threshold is used for limitation, there is a possibility of missing the recognition of the page turning action. Therefore, in this scenario, only the preset horizontal movement threshold and the preset movement speed are used as the recognition conditions to avoid missing the recognition of the page turning action and ensure the accuracy of the page turning action recognition. Among them, the extraction of the gesture movement trajectory can take the center of the palm as the target point, that is, the gesture movement trajectory describes the trajectory of the movement of the center point of the palm. Of course, other points in the hand contour can also be selected as the target point according to the application requirements, and the embodiment of the present invention does not make specific limitations. The preset horizontal movement threshold, the preset movement speed threshold and the preset movement speed can be customized according to the specific application requirements, and the embodiment of the present invention does not make specific limitations.

[0104] In one embodiment of the present invention, for further explanation and limitation, after determining that the action recognition result is a page turning action, the method further includes:

[0105] Obtaining the layout content of the unfolded page of the paper medium;

[0106] When the layout content matches the interactive display data displayed on the display screen, the interactive display data displayed on the display screen is kept unchanged.

[0107] In an embodiment of the present invention, after a page turning action is identified, the content of the unfolded page of the paper medium after the page turning action occurs is also collected, and the layout content of the current unfolded page is compared with the content of the unfolded page before the page turning action occurs. If the layout content of the unfolded page before and after the page turning action occurs is consistent, it indicates that a page turning action has occurred, but the actual paper has not turned a page, that is, the unfolded page has not changed. At this time, no page turning interaction is performed on the display screen, that is, the interactive display data displayed on the display screen before the page turning action occurs remains unchanged. By collecting and comparing the layout content of the unfolded page, it is possible to avoid misidentification of the page turning action when the user generates a page turning action but the paper medium does not actually turn a page, thereby further improving the accuracy of page turning action recognition and scene adaptability.

[0108] In an embodiment of the present invention, for further explanation and limitation, the method further includes:

[0109] When the action recognition result is a page turning action, determining the direction of the horizontal moving distance;

[0110] The page turning direction is determined according to the direction of the horizontal moving distance.

[0111] In an embodiment of the present invention, the interactive content of the display screen includes, in addition to the page turning action and interactive display data, the page turning direction. The page turning direction can be determined based on the direction of the horizontal moving distance of the gesture movement trajectory, with an image that meets the preset page turning gesture conditions as the initial frame image, and the coordinates of the target point in the initial frame image as the coordinate zero point. If the coordinate of the horizontal moving distance is a negative value, it indicates that the page turning gesture moves from right to left. According to the most common page turning direction, the current page turning action can be determined to be backward page turning; if the coordinate of the horizontal moving distance is a positive value, it indicates that the page turning gesture moves from left to right. The current page turning action can be determined to be forward page turning. By identifying the page turning direction, the page turning interactive animation in the display screen can be highly matched with the actual page turning action, thereby improving the realism of the paper screen interaction and thus improving the user experience.

[0112] In one embodiment of the present invention, for further explanation and limitation, the step of performing hand contour detection on the hand image to obtain hand contour information includes:

[0113] Performing blur denoising and smoothing filtering processing on the hand image to obtain a preprocessed hand image;

[0114] Converting the preprocessed hand image to an HSV color space, performing threshold segmentation on the preprocessed hand image according to a preset skin color range in the HSV color space, and obtaining a hand skin region image;

[0115] Performing morphological operation denoising on the hand skin area image to obtain a denoised hand skin area image;

[0116] The finger contour and palm contour target detection is performed based on the denoised hand skin area image to obtain hand contour information.

[0117] In an embodiment of the present invention, the hand contour detection process of a hand image includes three parts: image preprocessing, hand detection, and contour detection. Specifically, the image preprocessing is to convert each frame of the hand image into a grayscale image, perform Gaussian blur denoising on the grayscale image, and then perform smoothing filtering processing to obtain a hand image that is more conducive to detection (preprocessed hand image). Specifically, the hand detection is to convert the hand image into the HSV (Hue, Saturation, Value) color space, perform image segmentation on the skin area in the image according to the preset skin color range, and use morphological operations to remove the noise of the skin area obtained by segmentation to obtain a denoised hand skin area image. Contour detection specifically includes: a. Find all contours from the denoised hand skin area image; b. Ignore small contours from all the contours found; c. Find a curve that approximates a polygon and draws a circumscribed rectangle; d. Draw a convex hull, detect convex defects, ignore shallow defects, and find the coordinates of each finger and the coordinates of the palm.

[0118] In one embodiment of the present invention, for further explanation and limitation, the step of calculating the gesture movement trajectory according to the finger contour information corresponding to the multiple frames of the hand images includes:

[0119] Taking the hand image whose finger curvature satisfies the preset page-turning gesture condition as the starting frame image, calculating the hand position difference coordinates of each adjacent frame image based on the finger contour information of each adjacent frame image after the starting frame image, and creating a time window;

[0120] If within the time window, the hand position difference coordinates represent that the hand movement has stopped, then the gesture movement trajectory is calculated based on all the hand position difference coordinates between the start frame image and the end frame image;

[0121] If the hand position difference coordinates do not indicate that the hand movement has stopped at the end of the time window, the action recognition result is determined to be a non-page turning action, and the calculation of the hand position difference coordinates of adjacent frame images is stopped.

[0122] In the embodiment of the present invention, the hand position difference coordinates are the difference between the position coordinates of the same target point in adjacent frame images, which are used to characterize whether the hand is moving. Among them, the value of the hand position difference coordinates can be zero, or other values ​​close to zero, which is not specifically limited in the embodiment of the present invention. The image that meets the preset page turning gesture condition is used as the initial frame image, and the image with zero or close to zero displacement of the target point between two adjacent frame images is used as the termination frame image. The coordinate points of the target point in all frame images between the initial frame image and the termination frame image are connected to form the gesture movement trajectory. In view of the situation where the hand stays for a long time during the movement and no complete page turning action occurs, a time window is created at the same time as the gesture movement trajectory is collected. If the complete page turning action is completed before the end of this time window, that is, the termination frame image is recognized, the action recognition can be further performed based on the gesture movement trajectory. If the complete page turning action is not collected before the end of this time window, that is, the termination frame image is not recognized, it is considered that the current gesture movement trajectory is not a book turning action, and the gesture movement trajectory is stopped. Among them, the length of the time window can be 2s, or it can be customized according to actual needs, which is not specifically limited in the embodiment of the present invention.

[0123] In one example, the center point of the palm is taken as the target point, and the position coordinate changes of this target point in the continuous frame images are extracted. The coordinates of the target point in the initial frame image are set to (x1, y1), and the coordinates in the final frame image are set to (x2, y2). The time between the initial frame image and the final frame image is t. The horizontal movement distance is calculated to be dx=x2-x1, the vertical movement distance is dy=y2-y1, and the movement speed is S=dx / t. At the preset horizontal movement threshold X min = 100 pixels, preset moving speed threshold Y max =50 pixels, preset speed moving distance Speed min =150 pixels / second, when dx>X min ,dy<Y max , and S>Speed min If dx is greater than 0, it is determined to be a page turning action. If dx is less than 0, it is determined to be a page turning action.

[0124] The present invention provides an image-based paper-screen interaction method. In an embodiment of the present invention, when the interactive display data corresponding to the paper medium is displayed on a display screen, an image of the area where the paper medium is located is collected, and target detection is performed on the image; when the target detection result includes a hand image, hand contour information and finger curvature information are extracted from the hand image; when the finger curvature information meets the preset page-turning gesture condition, the gesture movement trajectory is calculated based on the hand contour information corresponding to multiple frames of hand images, and action recognition is performed based on the gesture movement trajectory; when the action recognition result is a page-turning action, a synchronous page-turning animation is displayed on the display screen based on the page-turning direction in the action recognition result, and the interactive display data of the page that is unfolded after the page is turned. The recognition starts from the time when the page-turning action is generated, which greatly advances the response time of the page-turning recognition, and at the same time, ensures the accuracy of the page-turning action recognition, thereby effectively improving the timeliness of the paper-screen interaction.

[0125] Furthermore, as a response to the above Figure 1 The implementation of the method shown in the figure, the embodiment of the present invention provides an image-based paper-screen interactive device, such as Figure 3 As shown, the device comprises:

[0126] The detection module 31 is used to collect an image of the area where the paper medium is located and perform target detection on the image when the display screen displays interactive display data corresponding to the paper medium;

[0127] An extraction module 32, configured to extract hand contour information and finger curvature information from the hand image when the target detection result includes the hand image;

[0128] The recognition module 33 is used to calculate the gesture movement trajectory according to the hand contour information corresponding to the multiple frames of the hand images when the finger curvature information meets the preset page turning gesture condition, and perform action recognition according to the gesture movement trajectory;

[0129] The interactive display module 34 is used to display a synchronous page turning animation and interactive display data of the unfolded page after page turning on the display screen according to the page turning direction in the action recognition result when the action recognition result is a page turning action.

[0130] Furthermore, the extraction module 32 includes:

[0131] a detection unit, configured to perform hand contour detection on the hand image to obtain hand contour information, wherein the hand contour information includes palm coordinate information, the number of fingers and key point coordinate information of each finger, and the key point coordinate information includes at least two of fingertip coordinates, fingertip joint coordinates, interphalangeal joint coordinates, and metacarpophalangeal joint coordinates;

[0132] a first calculation unit, for calculating, for each finger, coordinate information of at least one bending identification segment according to the key point coordinate information, wherein the bending identification segment includes a first identification segment between a fingertip and a fingertip joint, a second identification segment between a fingertip joint and an interphalangeal joint, and a third identification segment between an interphalangeal joint and a metacarpophalangeal joint;

[0133] The second calculation unit is used to calculate the finger curvature of each finger according to the coordinate information of at least one bending identification segment of the finger and the palm coordinate information to obtain the finger curvature information.

[0134] Furthermore, the device also includes:

[0135] The identification module 33 is further used to identify the bending state of the fingers one by one, and if the curvature of at least one bending identification segment of the finger is greater than the corresponding preset curvature threshold, the finger is determined to be a bent finger;

[0136] The determination module is used to determine that the finger bending degree meets the preset page turning gesture condition when the number of the bent fingers is greater than a preset number threshold.

[0137] Furthermore, the identification module 33 includes:

[0138] A first determining unit is used to determine that the action recognition result is a page turning action when the horizontal movement distance is greater than a preset horizontal movement threshold and the movement speed is greater than a preset movement speed threshold, wherein the gesture movement trajectory includes the horizontal movement distance and the movement speed; or

[0139] The second determination unit is used to determine that the action recognition result is a page turning action when the horizontal movement distance is greater than a preset horizontal movement threshold, if the vertical movement distance is less than the preset vertical movement threshold, and the movement speed is greater than the preset movement speed threshold, wherein the gesture movement trajectory includes the horizontal movement distance, the vertical movement distance and the movement speed.

[0140] Furthermore, the device also includes:

[0141] An acquisition module, used to acquire the layout content of the unfolded page of the paper medium;

[0142] The interactive maintaining module is used to maintain the interactive display data displayed on the display screen unchanged when the layout content matches the interactive display data displayed on the display screen.

[0143] Furthermore, the device also includes:

[0144] A first determination module, used for determining the direction of the horizontal moving distance when the action recognition result is a page turning action;

[0145] The second determining module is used to determine the page turning direction according to the direction of the horizontal moving distance.

[0146] Furthermore, in a specific application scenario, the detection unit is specifically used to perform blur denoising and smoothing filtering on the hand image to obtain a preprocessed hand image;

[0147] Converting the preprocessed hand image to an HSV color space, performing threshold segmentation on the preprocessed hand image according to a preset skin color range in the HSV color space, and obtaining a hand skin region image;

[0148] Performing morphological operation denoising on the hand skin area image to obtain a denoised hand skin area image;

[0149] The finger contour and palm contour target detection is performed based on the denoised hand skin area image to obtain hand contour information.

[0150] Furthermore, the identification module 33 includes:

[0151] A third calculation unit is used to use the hand image whose finger curvature satisfies the preset page turning gesture condition as the starting frame image, calculate the hand position difference coordinates of each adjacent frame image according to the finger contour information of each adjacent frame image after the starting frame image, and create a time window, wherein the hand position difference coordinates are used to indicate whether the hand is moving;

[0152] a fourth calculation unit, configured to calculate a gesture movement trajectory according to all hand position difference coordinates between the start frame image and the end frame image if the hand position difference coordinates indicate that the hand movement has stopped within the time window;

[0153] The fifth determination unit is used to determine that the action recognition result is a non-page turning action and stop calculating the hand position difference coordinates of adjacent frame images if the hand position difference coordinates do not represent the cessation of hand movement at the end of the time window.

[0154] The present invention provides an image-based paper-screen interactive device. The embodiment of the present invention collects an image of the area where the paper medium is located and performs target detection on the image when the interactive display data corresponding to the paper medium is displayed on the display screen; when the target detection result includes a hand image, the hand contour information and finger curvature information are extracted from the hand image; when the finger curvature information meets the preset page-turning gesture condition, the gesture movement trajectory is calculated based on the hand contour information corresponding to multiple frames of hand images, and action recognition is performed based on the gesture movement trajectory; when the action recognition result is a page-turning action, the page-turning direction in the action recognition result is displayed on the display screen according to the synchronous page-turning animation, and the interactive display data of the page after the page is turned. The recognition starts from the time the page-turning action is generated, which greatly advances the response time of the page-turning recognition, and at the same time, ensures the accuracy of the page-turning action recognition, thereby effectively improving the timeliness of the paper-screen interaction.

[0155] According to an embodiment of the present invention, a storage medium is provided, wherein the storage medium stores at least one executable instruction, and the computer executable instruction can execute the image-based paper-screen interaction method in any of the above method embodiments.

[0156] Figure 4 A schematic diagram of the structure of a terminal provided according to an embodiment of the present invention is shown. The specific embodiment of the present invention does not limit the specific implementation of the terminal.

[0157] like Figure 4 As shown, the terminal may include: a processor (processor) 402 , a communication interface (Communications Interface) 404 , a memory (memory) 406 , and a communication bus 408 .

[0158] The processor 402 , the communication interface 404 , and the memory 406 communicate with each other via a communication bus 408 .

[0159] The communication interface 404 is used for network communication with other devices such as a client or other servers.

[0160] The processor 402 is used to execute the program 410, and specifically can execute the relevant steps in the above-mentioned image-based paper-screen interaction method embodiment.

[0161] Specifically, the program 410 may include program codes, which include computer operation instructions.

[0162] The processor 402 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiment of the present invention. The one or more processors included in the terminal may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0163] The memory 406 is used to store the program 410. The memory 406 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0164] The program 410 may be specifically configured to enable the processor 402 to perform the following operations:

[0165] When the display screen displays interactive display data corresponding to the paper medium, an image of the area where the paper medium is located is collected, and target detection is performed on the image;

[0166] When the target detection result includes a hand image, extracting hand contour information and finger curvature information from the hand image;

[0167] When the finger curvature information satisfies a preset page-turning gesture condition, the gesture movement trajectory is calculated according to the hand contour information corresponding to the multiple frames of the hand images, and the action recognition is performed according to the gesture movement trajectory;

[0168] When the action recognition result is a page turning action, a synchronous page turning animation and interactive display data of the unfolded page after turning the page are displayed on the display screen according to the page turning direction in the action recognition result.

[0169] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0170] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An image-based paper-screen interaction method, characterized in that: include: When the display screen displays interactive display data corresponding to the paper medium, an image of the area where the paper medium is located is collected, and target detection is performed on the image; When the target detection result includes a hand image, extracting hand contour information and finger curvature information from the hand image; When the finger curvature information satisfies a preset page-turning gesture condition, the gesture movement trajectory is calculated according to the hand contour information corresponding to the multiple frames of the hand images, and the action recognition is performed according to the gesture movement trajectory; When the action recognition result is a page turning action, a synchronous page turning animation and interactive display data of the unfolded page after turning the page are displayed on the display screen according to the page turning direction in the action recognition result.

2. The method according to claim 1, characterized in that The step of extracting the hand contour information and the finger curvature information from the hand image includes: Performing hand contour detection on the hand image to obtain hand contour information, wherein the hand contour information includes palm coordinate information, the number of fingers and key point coordinate information of each finger, and the key point coordinate information includes at least two of fingertip coordinates, fingertip joint coordinates, interphalangeal joint coordinates, and metacarpophalangeal joint coordinates; For each finger, calculate the coordinate information of at least one bending identification segment according to the key point coordinate information, wherein the bending identification segment includes a first identification segment between the fingertip and the fingertip joint, a second identification segment between the fingertip joint and the interphalangeal joint, and a third identification segment between the interphalangeal joint and the metacarpophalangeal joint; The finger curvature of each finger is calculated according to the coordinate information of at least one bending identification segment of the finger and the palm coordinate information to obtain the finger curvature information.

3. The method according to claim 2, characterized in that The finger curvature information includes the curvature of at least one curvature identification segment corresponding to different fingers, and different curvature identification segments correspond to different preset curvature thresholds; After extracting the hand contour information and the finger curvature information from the hand image, the method further includes: Identify the bending state of the fingers one by one, and if the curvature of at least one bending identification segment of the finger is greater than the corresponding preset curvature threshold, determine that the finger is a bent finger; When the number of bent fingers is greater than a preset number threshold, it is determined that the finger bending degree meets the preset page turning gesture condition.

4. The method according to claim 1, characterized in that The performing action recognition according to the gesture movement trajectory includes: When the horizontal movement distance is greater than a preset horizontal movement threshold and the movement speed is greater than a preset movement speed threshold, the action recognition result is determined to be a page turning action, wherein the gesture movement trajectory includes the horizontal movement distance and the movement speed; or, When the horizontal movement distance is greater than the preset horizontal movement threshold, if the vertical movement distance is less than the preset vertical movement threshold, and the movement speed is greater than the preset movement speed threshold, the action recognition result is determined to be a page turning action, wherein the gesture movement trajectory includes the horizontal movement distance, the vertical movement distance and the movement speed.

5. The method according to claim 4, characterized in that After determining that the action recognition result is a page turning action, the method further includes: Obtaining the layout content of the unfolded page of the paper medium; When the layout content matches the interactive display data displayed on the display screen, the interactive display data displayed on the display screen is kept unchanged.

6. The method according to claim 4, characterized in that The method further comprises: When the action recognition result is a page turning action, determining the direction of the horizontal moving distance; The page turning direction is determined according to the direction of the horizontal moving distance.

7. The method according to claim 2, characterized in that The performing hand contour detection on the hand image to obtain hand contour information includes: Performing blur denoising and smoothing filtering processing on the hand image to obtain a preprocessed hand image; Converting the preprocessed hand image to an HSV color space, performing threshold segmentation on the preprocessed hand image according to a preset skin color range in the HSV color space, and obtaining a hand skin region image; Performing morphological operation denoising on the hand skin area image to obtain a denoised hand skin area image; The finger contour and palm contour target detection is performed based on the denoised hand skin area image to obtain hand contour information.

8. The method according to any one of claims 1 to 7, characterized in that The step of calculating the gesture movement trajectory based on the hand contour information corresponding to the multiple frames of hand images includes: The hand image whose finger curvature satisfies the preset page-turning gesture condition is taken as the starting frame image, and the hand position difference coordinates of each adjacent frame image are calculated according to the finger contour information of each adjacent frame image after the starting frame image, and a time window is created, wherein the hand position difference coordinates are used to characterize whether the hand is moving; If within the time window, the hand position difference coordinates represent that the hand movement has stopped, then the gesture movement trajectory is calculated based on all the hand position difference coordinates between the start frame image and the end frame image; If the hand position difference coordinates do not indicate that the hand movement has stopped at the end of the time window, the action recognition result is determined to be a non-page turning action, and the calculation of the hand position difference coordinates of adjacent frame images is stopped.

9. An image-based paper-screen interactive device, characterized in that: include: A detection module, used to collect an image of the area where the paper medium is located and perform target detection on the image when the display screen displays interactive display data corresponding to the paper medium; An extraction module, used for extracting hand contour information and finger curvature information from the hand image when the target detection result includes the hand image; A recognition module, configured to calculate a gesture movement trajectory based on hand contour information corresponding to a plurality of frames of hand images when the finger curvature information satisfies a preset page-turning gesture condition, and perform action recognition based on the gesture movement trajectory; The interactive display module is used to display a synchronous page turning animation and interactive display data of the unfolded page after turning the page on the display screen according to the page turning direction in the action recognition result when the action recognition result is a page turning action.

10. A storage medium storing at least one executable instruction, wherein the executable instruction enables a processor to execute operations corresponding to the image-based paper-screen interaction method as described in any one of claims 1-8.

11. A terminal, comprising: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the image-based paper-screen interaction method as described in any one of claims 1-8.

Images