Data input method, device, storage medium, and program product

Abstract

The application discloses a data input method and device, a storage medium and a program product. It relates to the field of meta universe, and the method comprises the following steps: in the case that a target object inputs data to an automatic teller machine according to a virtual interface, image acquisition is performed on the hand of the target object to obtain a hand gesture image; the hand boundary in the hand gesture image is calculated to obtain target boundary data; the coordinates of the fingertips of the target object are determined based on the target boundary data; and the data input by the target object to the automatic teller machine is determined based on the coordinates of the fingertips of the target object and the coordinates of the virtual interface. Through the application, the problem that the target object inputs data by clicking physical keys on the automatic teller machine in the process of handling financial business, and the data leakage risk is high, is solved.

Description

Technical Field

[0001] This application relates to the metaverse field, and more specifically, to a data input method, apparatus, storage medium, and program product. Background Technology

[0002] Automated teller machines, also known as ATMs, are highly sophisticated electromechanical devices that enable self-service financial transactions using bank cards. They can handle services such as withdrawals, deposits, transfers, balance inquiries, and password changes.

[0003] In typical usage scenarios, after inserting their bank card, customers select the desired transaction on the ATM's touchscreen or physical buttons, and then enter their PIN at the keypad for authentication. However, the data entered by customers on the physical keypad (e.g., their PIN) is easily observed by cameras or bystanders, leading to a high risk of user privacy breaches. Furthermore, if customers check their balance, the account balance is displayed on a large screen, and the discovery of large deposits by onlookers increases the customer's risk.

[0004] There is currently no effective solution to the problem of high data leakage risk when target individuals input data by clicking physical buttons on ATMs during financial transactions. Summary of the Invention

[0005] The main purpose of this application is to provide a data input method, device, storage medium, and program product to solve the problem of high data leakage risk when target objects input data by clicking physical buttons on ATMs during financial transactions.

[0006] To achieve the above objectives, according to one aspect of this application, a data input method is provided. The method includes: when a target object inputs data into an ATM via a virtual interface, acquiring an image of the target object's hand to obtain a gesture image; calculating the hand boundary in the gesture image to obtain target boundary data; determining the fingertip coordinates of the target object based on the target boundary data; and determining the data input by the target object into the ATM based on the fingertip coordinates of the target object and the coordinates of the virtual interface.

[0007] Further, calculating the hand boundary in the gesture image to obtain target boundary data includes: dividing the gesture image into a grid of a preset size to obtain M grids, where M is a positive integer; calculating the probability that each grid contains the center of the hand of the target object to obtain M probability values; extracting the grid associated with the highest probability value among the M probability values ​​to obtain the target grid; and determining the target boundary data based on the target grid.

[0008] Further, determining the target boundary data based on the target grid includes: when the probability value associated with the target grid is greater than a preset probability threshold, determining the hand coordinates in the gesture image based on the coordinates of the target grid, wherein the hand coordinates include at least one of the following: hand center coordinates, hand width, and hand height; and determining the target boundary data based on the hand coordinates.

[0009] Further, determining the hand coordinates in the gesture image based on the coordinates of the target grid includes: determining the center coordinates of the hand based on the coordinates of the target grid and the preset size; scaling the preset size based on a preset scaling factor to obtain the hand width and the hand height; and obtaining the hand coordinates based on the center coordinates of the hand, the hand width, and the hand height.

[0010] Further, determining the fingertip coordinates of the target object based on the target boundary data includes: cropping the gesture image based on the target boundary data to obtain a target region image, wherein the target region image includes the hand of the target object; inputting the target region image into a target convolutional neural network to obtain the fingertip coordinates of the target object, wherein the target convolutional neural network includes: a global average pooling layer and a fully connected layer, wherein the global average pooling layer is used to compress the spatial dimension of the target region image, and the fully connected layer is used to determine the fingertip coordinates of the target object.

[0011] Further, based on the target boundary data, the gesture image is cropped to obtain a target region image, including: determining the cropping frame size based on the target boundary data, wherein the height of the cropping frame size is greater than the hand height in the target boundary data, and the width of the cropping frame size is greater than the hand width in the target boundary data; cropping the gesture image based on the cropping frame size to obtain the target region image.

[0012] Further, based on the fingertip coordinates of the target object and the coordinates of the virtual interface, the data input by the target object to the ATM is determined, including: based on the coordinates of the virtual interface, determining the coordinates of each virtual button area in the virtual interface to obtain N button coordinates, where N is a positive integer; detecting whether the fingertip coordinates of the target object coincide with any of the button coordinates to obtain a detection result; if the detection result indicates that the fingertip coordinates of the target object coincide with any of the button coordinates, determining the data input by the target object to the ATM based on the button coordinates.

[0013] To achieve the above objectives, according to another aspect of this application, a data input device is provided. The device includes: a data acquisition unit for acquiring an image of the target object's hand when the target object inputs data into an ATM via a virtual interface, thereby obtaining a gesture image; a calculation unit for calculating the hand boundary in the gesture image, thereby obtaining target boundary data; a first determination unit for determining the fingertip coordinates of the target object based on the target boundary data; and a second determination unit for determining the data input by the target object into the ATM based on the fingertip coordinates of the target object and the coordinates of the virtual interface.

[0014] Further, the calculation unit includes: a division subunit, used to divide the gesture image into a grid of a preset size to obtain M grids, where M is a positive integer; a calculation subunit, used to calculate the probability that each grid contains the center of the hand of the target object to obtain M probability values; an extraction subunit, used to extract the grid associated with the highest probability value among the M probability values ​​to obtain the target grid; and a determination subunit, used to determine the target boundary data based on the target grid.

[0015] Further, the determining subunit includes: a first determining module, used to determine the hand coordinates in the gesture image based on the coordinates of the target grid when the probability value associated with the target grid is greater than a preset probability threshold, wherein the hand coordinates include at least one of the following: hand center coordinates, hand width, and hand height; and a second determining module, used to determine the target boundary data based on the hand coordinates.

[0016] Further, the first determining module includes: a first determining submodule, used to determine the center coordinates of the hand based on the coordinates of the target grid and the preset size; a first processing submodule, used to scale the preset size based on a preset scaling factor to obtain the hand width and the hand height; and a second processing submodule, used to obtain the hand coordinates based on the center coordinates of the hand, the hand width, and the hand height.

[0017] Further, the first determining unit includes: a cropping subunit, used to crop the gesture image based on the target boundary data to obtain a target region image, wherein the target region image includes the hand of the target object; and a processing subunit, used to input the target region image into a target convolutional neural network to obtain the fingertip coordinates of the target object, wherein the target convolutional neural network includes: a global average pooling layer and a fully connected layer, wherein the global average pooling layer is used to compress the spatial dimension of the target region image, and the fully connected layer is used to determine the fingertip coordinates of the target object.

[0018] Furthermore, the cropping subunit includes: a third determining module, used to determine the cropping frame size based on the target boundary data, wherein the height of the cropping frame size is greater than the hand height in the target boundary data, and the width of the cropping frame size is greater than the hand width in the target boundary data; and a cropping module, used to crop the gesture image based on the cropping frame size to obtain the target region image.

[0019] Further, the second determining module includes: a second determining submodule, used to determine the coordinates of each virtual button area in the virtual interface based on the coordinates of the virtual interface, to obtain N button coordinates, where N is a positive integer; a detection submodule, used to detect whether the fingertip coordinates of the target object coincide with any of the button coordinates, and to obtain a detection result; and a third determining submodule, used to determine the data input by the target object to the ATM based on the button coordinates when the detection result indicates that the fingertip coordinates of the target object coincide with any of the button coordinates.

[0020] According to another aspect of this application, a computer-readable storage medium is provided, the computer-readable storage medium including a stored executable program, wherein, when the executable program is executed, it controls the device where the computer-readable storage medium is located to perform the data input method.

[0021] According to another aspect of this application, an electronic device is provided, comprising: a memory storing an executable program; and a processor for running the program, wherein the program executes the data input method during runtime.

[0022] According to another aspect of this application, a computer program product is provided, including computer instructions that, when executed by a processor, implement the steps of the data input method.

[0023] This application employs the following method: when a target inputs data into an ATM via a virtual interface, an image of the target's hand is captured to obtain a gesture image; the hand boundary in the gesture image is calculated to obtain target boundary data; based on the target boundary data, the fingertip coordinates of the target are determined; and based on the fingertip coordinates and the coordinates of the virtual interface, the data input by the target into the ATM is determined. This solves the technical problem of high data leakage risk when a target inputs data by clicking physical buttons on an ATM during financial transactions. In this application, the ATM receives data input by the target through a virtual interface, avoiding the risk of data theft that occurs when clicking physical buttons, thus achieving the technical effect of improving the data security of ATM transactions. Attached Figure Description

[0024] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0025] Figure 1 A hardware structure block diagram of a computer terminal for implementing a data input method is shown.

[0026] Figure 2 This is a flowchart of a data input method provided according to an embodiment of this application;

[0027] Figure 3 This is a structural diagram of the data input system provided according to an embodiment of this application;

[0028] Figure 4 This is a flowchart of a hand recognition module provided according to an embodiment of this application;

[0029] Figure 5 This is a schematic diagram of a data input device provided according to an embodiment of this application;

[0030] Figure 6 This is a structural block diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0031] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0032] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0033] It should be noted that the data input method and apparatus in this application can be used in the fintech field when receiving customer input data, and can also be used in any field other than the fintech field when receiving customer input data. This application does not limit the application field of the data input method and apparatus.

[0034] First, some nouns or terms that appear in the description of the embodiments of this application shall be interpreted as follows:

[0035] CNN (Convolutional Neural Network): A convolutional neural network is a deep learning model specifically designed to process data with a grid structure (e.g., images, videos).

[0036] The target is a cross-platform framework for building real-time multimedia processing pipelines, primarily for tasks such as computer vision, speech processing, and action recognition.

[0037] ROI (Region of Interest): In image processing or computer vision, a region of interest is a specific area selected from the entire image for further analysis or processing.

[0038] It should be noted that the information collected in this application (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) are information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of this data all comply with relevant laws and standards, have taken necessary security measures, do not violate public order and good morals, and provide corresponding operation entry points for users to choose to authorize or refuse. For example, this system has interfaces with relevant users or organizations, providing users with corresponding operation entry points for users to choose to agree to or refuse automated decision results; if the user chooses to refuse, the process proceeds to the expert decision-making stage.

[0039] This invention can be applied to various software products, control systems, and client-side (including but not limited to: mobile clients, PCs, etc.) control systems of financial institutions to realize the business functions of financial institutions (including but not limited to: transfer, wealth management, funds, payment, account inquiry, advertising, recommendation, etc.).

[0040] Example 1

[0041] According to an embodiment of this application, a method embodiment for a data input method is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0042] The method embodiment provided in Embodiment 1 of this application can be executed on a mobile terminal, computer terminal, or similar computing device. Figure 1 A hardware block diagram of a computer terminal (or mobile device) for implementing a data input method is shown. Figure 1 As shown, computer terminal 10 (or mobile device) may include one or more ( Figure 1 The processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission device 106 for communication functions may also be included. In addition, it may include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0043] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10 (or mobile device). As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0044] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the data input method in this embodiment. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby realizing the above-mentioned data input method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0045] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0046] The display can be, for example, a touchscreen liquid crystal display (LCD), which allows the user to interact with the user interface of the computer terminal 10 (or mobile device).

[0047] Under the aforementioned operating environment, this application provides the following: Figure 2 The data input method is shown. Figure 2 This is a flowchart of the data input method according to Embodiment 1 of this application.

[0048] Step S201: When the target object inputs data into the ATM according to the virtual interface, the target object's hand image is captured to obtain a gesture image.

[0049] The aforementioned virtual interface may include a virtual interface displayed using augmented reality technology, which can show the target user the transaction information and virtual keyboard on the ATM. Before inputting data into the ATM, the target user may wear an external virtual device attached to the ATM, which may include, but is not limited to, AR (Augmented Reality) glasses or VR (Virtual Reality) glasses. The target user's hand may refer to their palm.

[0050] When the target object inputs data into the ATM through the virtual interface, the target object's hand movements can be captured by the image acquisition device (e.g., camera) on the ATM. The image information captured by the image acquisition device can be divided into frames to obtain the gesture movement image.

[0051] Step S202: Calculate the hand boundary in the gesture image to obtain target boundary data.

[0052] In this embodiment, feature vectors of the hand position in the gesture image can be extracted. Based on the feature vectors of the hand position, the hand boundary is determined to obtain target boundary data. For example, the gesture image can be divided into M grids, and feature vectors of each grid position can be extracted using a convolutional neural network. Based on the feature vectors of each grid position, the hand boundary is determined to obtain target boundary data. For example, the probability that each grid position contains the center of the hand can be calculated. Based on the probability that each grid position contains the center of the hand, the grid with the highest probability is identified as the center of the hand, and the target boundary data is determined based on the position of that grid.

[0053] Step S203: Determine the fingertip coordinates of the target object based on the target boundary data.

[0054] The fingertip coordinates of the target object can be obtained as the 3D coordinates of the fingertip. In this embodiment, the target area image can be obtained by cropping an image containing the hand of the target object from the gesture action image based on the target boundary data. Then, the 3D coordinates of multiple feature points in the target area image can be identified. Among these multiple feature points, the fingertip of the target object can be included. Thus, the fingertip coordinates of the target object can be obtained.

[0055] Step S204: Based on the fingertip coordinates of the target object and the coordinates of the virtual interface, determine the data input by the target object to the ATM.

[0056] In this embodiment, the data input by the target object to the ATM can be determined by whether the coordinates of the target object's fingertip coincide with the coordinates of the virtual buttons in the virtual interface.

[0057] Through the above steps, in this embodiment, the ATM receives data input by the target object through a virtual interface, avoiding the risk of data theft that occurs when data is input by clicking physical buttons on the ATM. This achieves the technical effect of improving the data security of ATM transactions. Furthermore, it solves the technical problem of high data leakage risk when the target object inputs data by clicking physical buttons on the ATM during financial transactions.

[0058] Optionally, in the data input method provided in this application embodiment, calculating the hand boundary in the gesture action image to obtain target boundary data includes: dividing the gesture action image into a grid of a preset size to obtain M grids, where M is a positive integer; calculating the probability that each grid contains the center of the hand of the target object to obtain M probability values; extracting the grid associated with the highest probability value among the M probability values ​​to obtain the target grid; and determining the target boundary data based on the target grid.

[0059] The aforementioned preset scale can include a pixel length of 8. In one optional example, the gesture image can be compressed to a standard size of 256×256, and then the image can be divided into a 32×32 grid, resulting in each grid having a pixel length of 8. Using a CNN as the backbone network, the input gesture image is progressively transformed into a high-dimensional feature map, and then feature vectors are extracted from specific locations in the feature map (i.e., the grid positions of each grid).

[0060] In this embodiment, the probability that each grid contains the center of the target object's hand can be calculated using a hand detection model, resulting in M ​​probability values. The grid associated with the highest probability value among the M probability values ​​is then extracted to obtain the target grid, and the target boundary data is determined based on the target grid.

[0061] For example, each grid cell can be traversed, and the confidence (i.e., probability) of a grid cell containing the center of the palm can be determined using the center point probability formula of the palm detection model. The formula is as follows: , where σ is the Sigmoid activation function ( W is a trainable weight vector (the dimension is the same as the number of channels in the feature map, typically 64). Let be the feature vector (also 64-dimensional) of the feature map at grid position (i, j), and b be the bias term. This formula maps high-dimensional features to probability values ​​in the range [0, 1]. The grid with the highest probability (i.e., the target grid) is determined as the center of the hand (e.g., the center of the palm). Based on the grid coordinates of the target grid, the target boundary data can be determined, achieving the technical effect of improving the accuracy of determining hand boundary data.

[0062] Optionally, in the data input method provided in this application embodiment, determining target boundary data based on the target grid includes: when the probability value associated with the target grid is greater than a preset probability threshold, determining the hand coordinates in the gesture action image based on the coordinates of the target grid, wherein the hand coordinates include at least one of the following: hand center coordinates, hand width, and hand height; and determining target boundary data based on the hand coordinates.

[0063] To improve the accuracy of hand boundary determination, for the target grid, it can also be determined whether the probability value of the target grid being the center of the hand is greater than a preset probability threshold. If the probability value associated with the target grid is greater than the preset probability threshold (e.g., 0.5), the hand coordinates in the gesture image are determined based on the coordinates of the target grid.

[0064] Optionally, in the data input method provided in this application embodiment, determining the hand coordinates in the gesture action image based on the coordinates of the target grid includes: determining the center coordinates of the hand based on the coordinates of the target grid and a preset size; scaling the preset size based on a preset scaling factor to obtain the hand width and hand height; and obtaining the hand coordinates based on the center coordinates of the hand, the hand width, and the hand height.

[0065] For the target mesh, the four parameters of the bounding box [Δx, Δy, Δw, Δh] can also be regressed simultaneously, where Δx and Δy are the offset ratios of the actual center of the hand relative to the top left corner of the mesh, and Δw and Δh are the logarithmic scaling factors of the bounding box width and height. If the center probability is greater than a threshold (e.g., 0.5), the bounding box of the hand is calculated using the following formula:

[0066]

[0067]

[0068]

[0069]

[0070] Where x and y are grid coordinates, , Equal to grid pixel length 8, calculation result , Let w be the center coordinates of the hand, w be the width of the hand, and h be the height of the hand. Then the bounding box of the hand is [ -w / 2, -h / 2, +w / 2, +h / 2).

[0071] The above formula is used to calculate the hand bounding box, which improves the accuracy of hand bounding box determination.

[0072] In one alternative example, for a hand detection model, the mathematical expression for the loss function during the model training phase is: ,in The center-point binary cross-entropy loss function, The bounding box Smooth L1 loss function is used. , Weight parameters (typical settings) =1, (5).

[0073]

[0074] Where N is the total number of grid cells (32×32=1024). The true probability is 1 = center of hand, 0 = non-center. Predict probabilities for the model (i.e.) ). Where M is the number of positive sample grids ( (grid size >0.5) For model prediction parameters, For actual parameters, "other" indicates other cases, i.e., |x|≥1.

[0075] Optionally, in the data input method provided in this application embodiment, determining the fingertip coordinates of the target object based on target boundary data includes: cropping the gesture image based on the target boundary data to obtain a target region image, wherein the target region image includes the hand of the target object; inputting the target region image into a target convolutional neural network to obtain the fingertip coordinates of the target object, wherein the target convolutional neural network includes: a global average pooling layer and a fully connected layer, wherein the global average pooling layer is used to compress the spatial dimension of the target region image, and the fully connected layer is used to determine the fingertip coordinates of the target object.

[0076] In this embodiment, the hand region containing the target object can be cropped from the gesture image based on the target boundary data to obtain the target region image. The 3D coordinates of multiple hand key points (e.g., 21 hand key points) can be extracted from the target region image using a target convolutional neural network (e.g., a hand key point detection model). This allows the 3D coordinates of the fingertips of the target object to be obtained. The multiple hand key points can be located at the joints, fingertips, and other positions of the hand.

[0077] For example, for a target region image, a CNN network can be used to convert the ROI image into a high-dimensional feature map of 8×8×128. The spatial dimension is then compressed using Global Average Pooling (GAP).

[0078]

[0079] Where v is a 128-dimensional vector, This is the feature vector of the feature map at grid position (i,j) (also 128-dimensional).

[0080] Finally, in this embodiment, v can be mapped to a 63-dimensional output using a fully connected regression layer (corresponding to the fully connected layer described above): , , For trainable weight vectors, The dimension is d×128. The dimension is 63×d (d is the hidden layer dimension, which can generally be set to 512). , This is the bias term. P represents the 3D coordinates of the 21 key points, expressed as follows:

[0081] .

[0082] In one alternative example, for the target convolutional neural network, the mathematical expression of the loss function used during the model training phase is:

[0083]

[0084] in, , , To predict coordinates, , , These are the actual coordinates.

[0085] Optionally, in the data input method provided in this application embodiment, cropping the gesture image based on target boundary data to obtain a target region image includes: determining the cropping box size based on the target boundary data, wherein the height of the cropping box size is greater than the hand height in the target boundary data, and the width of the cropping box size is greater than the hand width in the target boundary data; and cropping the gesture image based on the cropping box size to obtain the target region image.

[0086] For example, the ROI (corresponding to the target region image) can be cropped based on the hand's center coordinates, hand width, and hand height. During the cropping process, the center of the target region image can be aligned with the center of the standard coordinate system. Specifically, the side length of the cropping box can be calculated (appropriately enlarged to ensure the entire palm is included), and then the ROI region can be enlarged to a standard size of 256×256, with a scaling factor. For any point {x, y} within the ROI region, there is a mapping relationship:

[0087]

[0088] (128 is the center point of the 256×256 image, ensuring the ROI center is aligned with the standard coordinate system center). Perform the same operation on the RGB three channels to generate a 256×256×3 ROI image (corresponding to the target region image).

[0089] Optionally, in the data input method provided in this application embodiment, determining the data input by the target object to the ATM based on the fingertip coordinates of the target object and the coordinates of the virtual interface includes: determining the coordinates of each virtual button area in the virtual interface based on the coordinates of the virtual interface to obtain N button coordinates, where N is a positive integer; detecting whether the fingertip coordinates of the target object coincide with any button coordinate to obtain a detection result; and determining the data input by the target object to the ATM based on the button coordinates when the detection result indicates that the fingertip coordinates of the target object coincide with any button coordinate.

[0090] In this embodiment, the degree of overlap between the coordinates of the virtual interface and the fingertip coordinates of the target object can be used to determine whether a button has been pressed. For example, based on the fingertip coordinates of the target object and the coordinates of the virtual interface, the distance d from the fingertip to the virtual interface plane is calculated. If d is less than a threshold (e.g., 1cm), it is determined that the finger is close to the virtual interface, and then the next step of judgment is performed. Alternatively, the fingertip point can be projected onto the virtual interface plane and transformed to the corresponding local coordinate system to obtain the projection point Q. In the local coordinate system, it is determined whether Q is within the rectangular area of ​​a button. When a finger is detected within a button area and the finger has a speed of moving into the virtual interface plane, it is determined that the button has been pressed, and the data corresponding to the button is input into the ATM.

[0091] In this embodiment, when a customer conducts a transaction at an ATM, they can wear external AR glasses. The ATM communicates with the AR glasses to initiate a request, and a virtual interface is projected onto the customer's screen. The customer extends a finger, and the ATM's built-in camera recognizes the 3D coordinates of the fingertip. When the 3D coordinates of the fingertip coincide with a key area on the virtual keyboard, that key is highlighted, providing visual feedback. The customer then makes a tap gesture, and the ATM confirms the input. The ATM device primarily consists of an integrated device with a built-in camera, external AR glasses, and internal software including a hand recognition module.

[0092] An optional data input system can be used to execute the data input method provided in this embodiment. Figure 3 This is a structural diagram of the data input system provided according to an embodiment of this application, such as... Figure 3As shown, the system includes a hand recognition module and an ATM machine. The hand recognition module is primarily responsible for recognizing customer gestures and converting them into specific system operations. When the system receives an image signal from the camera, it automatically activates the hand recognition module, which performs real-time conversion of the customer's gestures during the transaction. Specifically, the image is input into a model within the target cross-platform framework. Through calculation, the input image is converted into 3D coordinates, and the movement trajectory of the hand is identified by the displacement of these 3D coordinates. When the hand's movement coincides with the coordinates of the virtual interface, it is considered that an operation has been performed.

[0093] Figure 4 This is a flowchart of a hand recognition module provided according to an embodiment of this application, such as... Figure 4 As shown, the captured video can be segmented into frames and converted into a high-dimensional feature map through CNN. The palm detection model calculates the palm bounding box, and the hand key point detection model outputs the 3D coordinates of 21 key points of the palm. By calculating the degree of overlap between the coordinates of the virtual interface and the coordinates of the fingers, it can be determined whether the customer has pressed a certain button.

[0094] This embodiment accurately identifies customer gestures, ensuring that only the customer can see the virtual interface through AR glasses. This achieves the technical effects of improving data recognition speed, enhancing customer experience, and avoiding the risk of data being spied on.

[0095] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0096] Example 2

[0097] This application also provides a data input device. It should be noted that the data input device of this application can be used to execute the data input method provided in this application. The data input device provided in this application is described below.

[0098] According to an embodiment of this application, an apparatus for implementing the above-described data input method is also provided, such as... Figure 5 As shown, the device includes: a data acquisition unit 51, a calculation unit 52, a first determination unit 53, and a second determination unit 54.

[0099] The acquisition unit 51 is used to acquire images of the target object's hands and obtain gesture images when the target object inputs data into the ATM according to the virtual interface.

[0100] The calculation unit 52 is used to calculate the hand boundary in the gesture action image to obtain target boundary data;

[0101] The first determining unit 53 is used to determine the fingertip coordinates of the target object based on the target boundary data;

[0102] The second determining unit 54 is used to determine the data input by the target object to the ATM based on the fingertip coordinates of the target object and the coordinates of the virtual interface.

[0103] To achieve the above objectives, according to another aspect of this application, a data input device is provided. This device includes: acquiring an image of the target object's hand via a collection unit 51 when the target object inputs data into an ATM through a virtual interface, obtaining a gesture image; calculating the hand boundary in the gesture image via a calculation unit 52 to obtain target boundary data; determining the fingertip coordinates of the target object based on the target boundary data via a first determination unit 53; and determining the data input by the target object into the ATM via a second determination unit 54 based on the fingertip coordinates of the target object and the coordinates of the virtual interface. This solves the technical problem of high data leakage risk when the target object inputs data by clicking physical buttons on the ATM during financial transactions. In this embodiment, the ATM receives the data input by the target object through a virtual interface, avoiding the risk of data theft that occurs when inputting data by clicking physical buttons, thereby achieving the technical effect of improving the data security of ATM transactions.

[0104] Optionally, in the data input device provided in this application embodiment, the calculation unit includes: a division subunit, used to divide the gesture action image into a grid of a preset size to obtain M grids, where M is a positive integer; a calculation subunit, used to calculate the probability that each grid contains the center of the hand of the target object to obtain M probability values; an extraction subunit, used to extract the grid associated with the highest probability value among the M probability values ​​to obtain the target grid; and a determination subunit, used to determine the target boundary data based on the target grid.

[0105] Optionally, in the data input device provided in this application embodiment, the determining subunit includes: a first determining module, used to determine the hand coordinates in the gesture action image based on the coordinates of the target grid when the probability value associated with the target grid is greater than a preset probability threshold, wherein the hand coordinates include at least one of the following: hand center coordinates, hand width, and hand height; and a second determining module, used to determine target boundary data based on the hand coordinates.

[0106] Optionally, in the data input device provided in the embodiments of this application, the first determining module includes: a first determining submodule, used to determine the center coordinates of the hand based on the coordinates of the target grid and a preset size; a first processing submodule, used to scale the preset size based on a preset scaling factor to obtain the hand width and hand height; and a second processing submodule, used to obtain the hand coordinates based on the center coordinates of the hand, the hand width, and the hand height.

[0107] Optionally, in the data input device provided in this application embodiment, the first determining unit includes: a cropping subunit, used to crop the gesture action image based on target boundary data to obtain a target region image, wherein the target region image includes the hand of the target object; and a processing subunit, used to input the target region image into a target convolutional neural network to obtain the fingertip coordinates of the target object, wherein the target convolutional neural network includes: a global average pooling layer and a fully connected layer, wherein the global average pooling layer is used to compress the spatial dimension of the target region image, and the fully connected layer is used to determine the fingertip coordinates of the target object.

[0108] Optionally, in the data input device provided in this application embodiment, the cropping subunit includes: a third determining module, used to determine the cropping frame size based on target boundary data, wherein the height in the cropping frame size is greater than the hand height in the target boundary data, and the width in the cropping frame size is greater than the hand width in the target boundary data; and a cropping module, used to crop the gesture image based on the cropping frame size to obtain a target region image.

[0109] Optionally, in the data input device provided in this application embodiment, the second determining module includes: a second determining submodule, used to determine the coordinates of each virtual button area in the virtual interface based on the coordinates of the virtual interface, to obtain N button coordinates, where N is a positive integer; a detection submodule, used to detect whether the fingertip coordinates of the target object coincide with any button coordinate, and to obtain a detection result; and a third determining submodule, used to determine the data input by the target object to the ATM based on the button coordinates when the detection result indicates that the fingertip coordinates of the target object coincide with any button coordinate.

[0110] It should be noted that the acquisition unit 51, calculation unit 52, first determination unit 53, and second determination unit 54 mentioned above correspond to steps S201 to S204 in Embodiment 1. Each unit and its corresponding step implement the same instance and application scenario, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules or units can be hardware or software components stored in memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n). The above modules can also be part of a device and run in the computer terminal 10 provided in Embodiment 1.

[0111] Example 3

[0112] Embodiments of this application may provide an electronic device. Figure 6 This is a structural block diagram of an electronic device according to an embodiment of this application. Figure 6 As shown, the electronic device may include: one or more ( Figure 6 (Only one is shown) Processor 602, memory 604, memory controller, and peripheral interface, wherein the peripheral interface is connected to the radio frequency module, audio module and display.

[0113] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the methods and apparatus in the embodiments of this application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby implementing the above-described methods. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0114] The processor can access the information and application programs stored in the memory via the transmission device to perform the following steps: when the target object inputs data into the ATM according to the virtual interface, the processor acquires an image of the target object's hand to obtain a gesture image; the processor calculates the hand boundary in the gesture image to obtain target boundary data; based on the target boundary data, the processor determines the fingertip coordinates of the target object; based on the fingertip coordinates of the target object and the coordinates of the virtual interface, the processor determines the data input by the target object into the ATM.

[0115] The processor can also call the information and application programs stored in the memory through the transmission device to perform the following steps: calculate the hand boundary in the gesture action image to obtain target boundary data, including: dividing the gesture action image into a grid of preset size to obtain M grids, where M is a positive integer; calculating the probability that each grid contains the center of the hand of the target object to obtain M probability values; extracting the grid associated with the highest probability value among the M probability values ​​to obtain the target grid; and determining the target boundary data based on the target grid.

[0116] The processor can also call the information and application stored in the memory through the transmission device to perform the following steps: determining target boundary data based on the target grid, including: when the probability value associated with the target grid is greater than a preset probability threshold, determining the hand coordinates in the gesture action image based on the coordinates of the target grid, wherein the hand coordinates include at least one of the following: hand center coordinates, hand width, and hand height; determining target boundary data based on the hand coordinates.

[0117] The processor can also call the information and application stored in the memory through the transmission device to perform the following steps: determining the hand coordinates in the gesture image based on the coordinates of the target grid, including: determining the center coordinates of the hand based on the coordinates of the target grid and the preset size; scaling the preset size based on the preset scaling factor to obtain the hand width and hand height; and obtaining the hand coordinates based on the center coordinates of the hand, the hand width, and the hand height.

[0118] The processor can also access information and applications stored in the memory via a transmission device to perform the following steps: determining the fingertip coordinates of the target object based on target boundary data, including: cropping the gesture image based on the target boundary data to obtain a target region image, wherein the target region image includes the hand of the target object; inputting the target region image into a target convolutional neural network to obtain the fingertip coordinates of the target object, wherein the target convolutional neural network includes: a global average pooling layer and a fully connected layer, wherein the global average pooling layer is used to compress the spatial dimension of the target region image, and the fully connected layer is used to determine the fingertip coordinates of the target object.

[0119] The processor can also access information and applications stored in the memory via a transmission device to perform the following steps: cropping the gesture image based on target boundary data to obtain a target region image, including: determining the cropping box size based on the target boundary data, wherein the height of the cropping box is greater than the hand height in the target boundary data, and the width of the cropping box is greater than the hand width in the target boundary data; cropping the gesture image based on the cropping box size to obtain the target region image.

[0120] The processor can also call the information and application programs stored in the memory through the transmission device to perform the following steps: determining the data input by the target object to the ATM based on the fingertip coordinates of the target object and the coordinates of the virtual interface, including: determining the coordinates of each virtual key area in the virtual interface based on the coordinates of the virtual interface, obtaining N key coordinates, where N is a positive integer; detecting whether the fingertip coordinates of the target object coincide with any key coordinate, and obtaining a detection result; if the detection result indicates that the fingertip coordinates of the target object coincide with any key coordinate, determining the data input by the target object to the ATM based on the key coordinate.

[0121] By using the embodiments of this application, the ATM receives data input by the target object through a virtual interface, avoiding the risk of data theft that can be easily committed by clicking physical buttons, thereby achieving the technical effect of improving the data security of ATM transactions.

[0122] Those skilled in the art will understand that Figure 6 The structure shown is for illustrative purposes only. Electronic devices can also be smartphones, tablets, handheld computers, mobile internet devices (MIDs), PADs, and other terminal devices. Figure 6 This does not limit the structure of the aforementioned electronic device. For example, electronic devices may also include components that are more... Figure 6 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 6 The different configurations shown.

[0123] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0124] Example 4

[0125] Embodiments of this application also provide a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the data input method provided in Embodiment 1.

[0126] Optionally, in this embodiment, the storage medium may be located in any computer terminal in a group of computer terminals in a computer network, or in any mobile terminal in a group of mobile terminals.

[0127] This application also provides a computer program product that, when executed on a data processing device, is suitable for performing data input method steps.

[0128] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0129] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0130] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0131] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0132] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0133] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0134] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A data entry method, characterized by, include: When the target object inputs data into the ATM through the virtual interface, the image of the target object's hand is captured to obtain a gesture image; Calculate the hand boundary in the gesture image to obtain target boundary data; Based on the target boundary data, determine the fingertip coordinates of the target object; Based on the fingertip coordinates of the target object and the coordinates of the virtual interface, the data input by the target object to the ATM is determined.

2. The data input method of claim 1, wherein, Calculate the hand boundary in the gesture image to obtain target boundary data, including: The gesture image is divided into a grid of a preset size, resulting in M ​​grids, where M is a positive integer; Calculate the probability that each grid contains the center of the target object's hand, and obtain M probability values; Extract the grid associated with the highest probability value among the M probability values ​​to obtain the target grid; Based on the target grid, the target boundary data is determined.

3. The data input method according to claim 2, characterized in that, Based on the target mesh, the target boundary data is determined, including: If the probability value associated with the target grid is greater than a preset probability threshold, the hand coordinates in the gesture image are determined based on the coordinates of the target grid, wherein the hand coordinates include at least one of the following: hand center coordinates, hand width, and hand height; The target boundary data is determined based on the hand coordinates.

4. The data input method according to claim 3, characterized in that, Determining the hand coordinates in the gesture image based on the coordinates of the target grid includes: Based on the coordinates of the target grid and the preset size, determine the coordinates of the hand center. The preset size is scaled based on a preset scaling factor to obtain the hand width and the hand height; The hand coordinates are obtained based on the hand center coordinates, the hand width, and the hand height.

5. The data input method according to claim 1, characterized in that, Based on the target boundary data, the fingertip coordinates of the target object are determined, including: Based on the target boundary data, the gesture image is cropped to obtain a target region image, wherein the target region image includes the hand of the target object; The target region image is input into a target convolutional neural network to obtain the fingertip coordinates of the target object. The target convolutional neural network includes a global average pooling layer and a fully connected layer. The global average pooling layer is used to compress the spatial dimension of the target region image, and the fully connected layer is used to determine the fingertip coordinates of the target object.

6. The data input method according to claim 4, characterized in that, Based on the target boundary data, the gesture image is cropped to obtain a target region image, including: Based on the target boundary data, the cropping frame size is determined, wherein the height of the cropping frame size is greater than the hand height in the target boundary data, and the width of the cropping frame size is greater than the hand width in the target boundary data; The gesture image is cropped based on the cropping frame size to obtain the target region image.

7. The data input method according to claim 1, characterized in that, Based on the fingertip coordinates of the target object and the coordinates of the virtual interface, the data input by the target object to the ATM is determined, including: Based on the coordinates of the virtual interface, the coordinates of each virtual button area in the virtual interface are determined, resulting in N button coordinates, where N is a positive integer; The detection result is obtained by detecting whether the fingertip coordinates of the target object coincide with the coordinates of any of the keys. If the detection result indicates that the fingertip coordinates of the target object coincide with the coordinates of any of the buttons, the data input by the target object to the ATM is determined based on the button coordinates.

8. A data input device, characterized in that, include: The acquisition unit is used to acquire images of the target object's hands when the target object inputs data into the ATM according to the virtual interface, and obtain gesture action images. The calculation unit is used to calculate the hand boundary in the gesture image to obtain target boundary data; The first determining unit is used to determine the fingertip coordinates of the target object based on the target boundary data; The second determining unit is used to determine the data input by the target object to the ATM based on the fingertip coordinates of the target object and the coordinates of the virtual interface.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored executable program, wherein, when the executable program is executed, it controls the device on which the computer-readable storage medium is located to perform the data input method according to any one of claims 1 to 7.

10. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the steps of the data input method according to any one of claims 1 to 7.

Images