Information processing method and device and electronic equipment

Abstract

The invention relates to an information processing method and device and electronic equipment, and relates to the field of education, and the method comprises the steps: obtaining a hand image, collected by a camera, of a target user; performing image correction on the hand image, and performing feature recognition on the corrected hand image to obtain hand feature information of the target user; performing feature processing on the hand feature information by using a convolutional neural network, and determining a fingertip position of the target user; the information recognition area corresponding to the fingertip position is determined, the to-be-scanned information in the information recognition area is processed according to a corresponding scanning function, and the scanning function comprises a word scanning translation function and / or a question scanning question answering function. The input efficiency can be improved, the user experience is enhanced, and meanwhile high practicability and wide application prospects are achieved.

Description

Technical Field

[0001] The present application relates to the field of computer vision and artificial intelligence technology, and specifically to an information processing method, device and electronic device. Background Art

[0002] With the advancement of technology, more and more software and learning machines have various learning functions to help children with their homework. However, the existing software and learning machines' functions such as search terms, text translation, and search questions mostly rely on keyboard input or touch screen operation. However, these input methods may not be convenient or fast enough in some cases. Therefore, it is necessary to develop a new information processing method to improve information processing efficiency and enhance user experience. Summary of the invention

[0003] In view of this, the present disclosure provides an information processing method, device and electronic device, the main purpose of which is to improve the problem that the current information processing method may not be convenient or fast enough in some cases.

[0004] In a first aspect, the present disclosure provides an information processing method, comprising:

[0005] Obtain a hand image of the target user captured by the camera;

[0006] Performing image correction on the hand image, and performing feature recognition on the corrected hand image to obtain hand feature information of the target user;

[0007] Using a convolutional neural network to perform feature processing on the hand feature information to determine the fingertip position of the target user;

[0008] Determine the information recognition area corresponding to the fingertip position, and process the information to be scanned in the information recognition area according to the corresponding scanning function, wherein the scanning function includes a word scanning translation function and / or a question scanning answering function.

[0009] In a second aspect, the present disclosure provides an information processing device, including:

[0010] An acquisition module, used to acquire a hand image of a target user captured by a camera;

[0011] A correction module, used to perform image correction on the hand image, and obtain hand feature information of the target user by performing feature recognition on the corrected hand image;

[0012] A processing module, used to perform feature processing on the hand feature information using a convolutional neural network to determine the fingertip position of the target user;

[0013] A determination module is used to determine the information recognition area corresponding to the fingertip position, and process the information to be scanned in the information recognition area according to the corresponding scanning function, wherein the scanning function includes a word scanning translation function and / or a question scanning answering function.

[0014] In a third aspect, the present disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the information processing method described in the first aspect.

[0015] In a fourth aspect, the present disclosure provides an electronic device, comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor implements the information processing method described in the first aspect when executing the computer program.

[0016] By means of the above technical scheme, the present disclosure provides an information processing method, device and electronic device. Compared with the current prior art, the present disclosure can obtain the hand image of the target user captured by the camera; perform image correction on the hand image, and obtain the hand feature information of the target user by performing feature recognition on the corrected hand image; perform feature processing on the hand feature information using a convolutional neural network to determine the fingertip position of the target user; determine the information recognition area corresponding to the fingertip position, and process the information to be scanned in the information recognition area according to the corresponding scanning function, and the scanning function includes a word scanning translation function and / or a question scanning and answering function. By adopting the above technical scheme, after determining the hand feature information by performing feature recognition on the hand image, the information to be scanned can be automatically processed based on the hand feature information, which can improve the information input efficiency under the corresponding scanning function, thereby improving the user experience.

[0017] The above description is only an overview of the technical solution of the present disclosure. In order to more clearly understand the technical means of the present disclosure, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present disclosure more obvious and easy to understand, the specific implementation methods of the present disclosure are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0019] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0020] Figure 1A flowchart of an information processing method provided by an embodiment of the present disclosure;

[0021] Figure 2 A flowchart of another information processing method provided by an embodiment of the present disclosure;

[0022] Figure 3 A schematic diagram of the structure of an information processing device provided by an embodiment of the present disclosure;

[0023] Figure 4 A schematic diagram of the structure of another information processing device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0024] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as limiting the present invention.

[0025] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0026] In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0027] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0028] In order to improve the problem that the existing software and learning machines in the prior art, such as search terms, text translation, and search title functions, mostly rely on keyboard input or touch screen operation, which may not be convenient or fast enough in some cases. The present disclosure provides an information processing method, such as Figure 1 As shown, the method includes:

[0029] Step 101: Acquire a hand image of a target user captured by a camera.

[0030] For the embodiments of the present disclosure, a camera may be used to capture real-time images of the target user's hands to obtain a hand image of the target user, wherein the target user may be a person entering the camera's shooting range, and the hand image may be an image that may include the complete hand area or an image that includes a partial hand area, such as a fingertip area, etc.

[0031] For the embodiments of the present disclosure, the execution subject may be an information scanning device, wherein the information scanning device may be of various types, such as a scanning watch, a scanner, etc., and no specific limitation is made to this.

[0032] Step 102: perform image correction on the hand image, and perform feature recognition on the corrected hand image to obtain hand feature information of the target user.

[0033] For the embodiments of the present disclosure, the camera may produce geometric distortion when capturing the hand image of the target user, and it is necessary to first perform image correction on the hand image of the target user, so as to more accurately identify the hand feature information of the target user. Among them, the hand feature information may be features of hand joint coordinates, joint connections, etc., and there is no specific limitation on this; the geometric distortion of the image refers to the error in the geometric position of the image point on the image plane, because the imaging system cannot make the image and the actual scene strictly meet the pinhole imaging type (or central projection relationship) within the entire field of view, causing the central projection ray to bend. Among them, the distortion can be divided into radial distortion and tangential distortion. When performing quantitative analysis on the image, the distorted image must first be accurately geometrically corrected. For the embodiments of the present disclosure, there can be multiple methods for image correction, and there is no specific limitation on this.

[0034] Step 103: Use a convolutional neural network to perform feature processing on the hand feature information to determine the fingertip position of the target user.

[0035] Among them, Convolutional Neural Networks (CNN) is a type of feedforward neural network that includes convolution calculations and has a deep structure. Due to the differences in hierarchical structure and filter size, Convolutional Neural Networks can capture high-level, intermediate and even low-level features, and can also compress information into a smaller size through a pooling mechanism; the feature processing process may include feature extraction, fusion, classification and other steps, which are not specifically limited.

[0036] For the embodiments of the present disclosure, when the convolutional neural network performs feature extraction on hand feature information, as a possible implementation method, the deep network convolution layer of the convolutional neural network can be used to extract the first hand feature in the hand feature information. Since the receptive field of the deep network convolution layer increases and the overlapping area between the receptive fields increases, the hand image can be compressed, so that the first hand feature obtained is all the feature points in the fingertip area of ​​the hand image.

[0037] As a possible implementation method, the shallow network convolution layer of the convolutional neural network can be used to extract the second hand features of the target user from the hand feature information. Since the receptive field of the shallow network convolution layer is small and the overlapping area of ​​the receptive field is also small, the captured second hand features can contain more detailed information, such as the color, texture, edge, and corners of the fingertips, and there is no specific limitation on this.

[0038] In the disclosed embodiments, when the convolutional neural network performs feature fusion on hand feature information, there may be multiple methods of feature fusion, such as early fusion, late fusion, etc., which are not specifically limited. The first hand feature and the second hand feature may be feature fused, and the result of the feature fusion may be feature classified, wherein there may be multiple methods of feature classification, such as using the L-Softmax Loss function to classify and output the features, which is not specifically limited. The fingertip position of the target user is extracted from the feature classification result, wherein the feature classification result may include the fingertip position coordinates of the target user.

[0039] For the embodiments of the present disclosure, the specific implementation process can be described as: using a convolutional neural network to perform feature processing on hand feature information to determine the fingertip position of the target user; using a deep network convolutional layer of the convolutional neural network to extract a first hand feature from the hand feature information, and using a shallow network convolutional layer of the convolutional neural network to extract a second hand feature from the hand feature information, the first hand feature is all feature points in the fingertip area of ​​the hand, and the second hand feature is the texture edge of the fingertip area of ​​the hand; performing feature fusion on the first hand feature and the second hand feature, and performing feature classification on the result of the feature fusion, and extracting the fingertip position of the target user from the feature classification result.

[0040] Step 104 : determine the information recognition area corresponding to the fingertip position, and process the information to be scanned in the information recognition area according to the corresponding scanning function.

[0041] For the disclosed embodiment, the fingertip position may be the image coordinates corresponding to the fingertip area in the hand image, and the information identification area may be the area that can be scanned and identified when the information scanning device uses the fingertip position as the scanning reference point. Among them, different information identification areas correspond to different information to be scanned, and the information to be scanned is the information to be scanned in the information identification area, which may include words, text, questions, etc., and there is no specific limitation on this. Different information to be scanned may correspond to different scanning functions, and the scanning functions may include, for example, a word scanning translation function and / or a question scanning question answering function.

[0042] Exemplarily, after the fingertip area corresponding to the fingertip position is determined as the information recognition area, the information scanning device can scan and identify the information to be scanned in the information recognition area. After determining the information category of the information to be scanned (such as English), the English in the information recognition area can be translated according to the corresponding scanning function (such as translation) of the information category of English.

[0043] In summary, according to an information processing method, device and electronic device provided by the present disclosure, compared with the current prior art, the present disclosure can obtain a hand image of a target user captured by a camera; perform image correction on the hand image, and obtain the hand feature information of the target user by performing feature recognition on the corrected hand image; perform feature processing on the hand feature information using a convolutional neural network to determine the fingertip position of the target user; determine the information recognition area corresponding to the fingertip position, and process the information to be scanned in the information recognition area according to the corresponding scanning function, and the scanning function includes a word scanning translation function and / or a question scanning and answering function. By adopting the above technical solution, after determining the hand feature information by performing feature recognition on the hand image, the information to be scanned can be automatically processed based on the hand feature information, which can improve the information input efficiency under the corresponding scanning function, thereby improving the user experience.

[0044] Further, as a refinement and extension of the above embodiment, in order to fully illustrate the specific implementation process of the method of the embodiment of the present disclosure, the embodiment of the present disclosure provides the following Figure 2 The specific method shown includes:

[0045] Step 201: Acquire a hand image of a target user captured by a camera.

[0046] For the implementation process of the embodiment of the present disclosure, please refer to the relevant description in step 101 of the embodiment, which will not be repeated here.

[0047] Step 202: calibrate the hand image by using a preset calibration device to obtain internal parameters and distortion parameters corresponding to the camera, remap the hand image based on the internal parameters, and reconstruct the grayscale of the remapped hand image based on the distortion parameters to obtain a corrected image corresponding to the hand image.

[0048] In the disclosed embodiment, before image correction is performed on the hand image, the collected hand image may be preprocessed to eliminate irrelevant information in the hand image, retain the area where the hand is located, enhance the detectability of the hand information, and simplify the data to the maximum extent, thereby improving the reliability of feature extraction, image segmentation, matching, and recognition. The preprocessing may include operations such as image cropping, graying, geometric transformation, and image enhancement, which are not specifically limited.

[0049] In specific application scenarios, when capturing the hand image of the target user, other image information may be introduced into the hand image, resulting in inaccurate recognition of the hand feature information. In the disclosed embodiment, the area where the hand is located in the hand image can be identified, and the hand image can be cropped to reduce interference caused by other image information. The other image information may be of multiple types, and there is no specific limitation on this.

[0050] In a specific application scenario, since the hand image is a color image, when processing the cropped hand image, it is often necessary to process the three channels in sequence, which will take a lot of time. Therefore, in order to improve the processing speed of the entire application system, it is necessary to grayscale the hand image to reduce the amount of data to be processed. There are many grayscale methods, such as weighted average method, component method, average method, etc., which are not specifically limited.

[0051] In specific application scenarios, image geometric transformation is also called image space transformation. The grayscale hand image can be processed through geometric transformations such as translation, transposition, mirroring, rotation, and scaling to correct the systematic errors of the hand image acquisition system and the random errors of the instrument position (imaging angle, perspective relationship, and even the lens itself). In addition, a grayscale interpolation algorithm is also required, because according to this transformation relationship, the pixels of the output image may be mapped to non-integer coordinates of the input image. Commonly used methods include nearest neighbor interpolation, bilinear interpolation, and bicubic interpolation, which are not specifically limited.

[0052] In specific application scenarios, in order to improve the visual effect of the image, for the application scenarios of hand images, the overall or local characteristics of the image are purposefully emphasized, the originally unclear hand image is made clear or the required features are emphasized, the differences between the features of different objects in the image are enlarged, and unnecessary features are suppressed, thereby improving the image quality, enriching the amount of information, and enhancing the image interpretation and recognition effect. The hand image that has undergone the above series of operations is subjected to image enhancement processing, wherein the image enhancement processing can use frequency domain method, space domain method, etc., and this embodiment does not make specific limitations on this.

[0053] For the disclosed embodiments, the preset calibration device is a tool for calibrating hand images in the camera that can be pre-set according to actual conditions, and no specific limitation is made to this. The camera may produce geometric distortion when collecting the hand image of the target user, and it is necessary to first perform image correction on the hand image of the target user so as to more accurately identify the hand feature information of the target user. The imaging process of the camera lens undergoes the conversion of four coordinate systems, namely the world coordinate system, the camera coordinate system, the image coordinate system, and the pixel coordinate system. The world coordinate system is also called the real coordinate system or the global coordinate system. It is the absolute coordinate of the objective world. It is a three-dimensional space coordinate system arbitrarily defined by the user. Generally, three-dimensional scenes are represented by the world coordinate system. The camera coordinate system is a three-dimensional rectangular coordinate system with the focusing center of the pinhole camera model as the origin and the lens optical axis as the Z axis. The X-axis and Y-axis are parallel to the x-axis and y-axis of the image coordinate system. The origin of the image coordinate system is the intersection of the lens optical axis and the imaging plane. The x-axis and y-axis are parallel to the X-axis and Y-axis of the camera coordinate system respectively. It is a plane rectangular coordinate system with units of millimeters. The pixel coordinate system is also called the computer image coordinate system. It is a plane rectangular coordinate system with pixels fixed on the image. Its origin is located in the upper left corner of the image, and the u-axis and v-axis are parallel to the x-axis and y-axis of the image coordinate system.

[0054] In the disclosed embodiment, the internal parameters and distortion parameters corresponding to the specified camera are determined based on the first conversion relationship between the world coordinate system and the camera coordinate system, the second conversion relationship between the camera coordinate system and the image coordinate system, the third conversion relationship between the image coordinate system and the pixel coordinate system, and the calibration information obtained by calibrating the distorted image with the camera calibration tool. The first conversion relationship includes: the conversion relationship between the world coordinate point in the world coordinate system and the camera coordinate in the camera coordinate system, the second conversion relationship includes: the conversion relationship between the camera coordinate point in the camera coordinate system and the image point in the image coordinate system. The third conversion relationship includes: the conversion relationship between the image coordinate system and the pixel coordinate system.

[0055] For the embodiments of the present disclosure, remapping is the process of placing pixels at a certain position in a hand image to a specified position in another image. There may be various methods for remapping, such as using the function remap in OpenCV to implement remapping based on internal parameters, and no specific limitation is made to this. There may be various methods for grayscale reconstruction, such as the nearest neighbor interpolation method, bilinear interpolation method, cubic convolution interpolation method, etc., and no specific limitation is made to this. By performing grayscale reconstruction on the remapped hand image, a corrected image corresponding to the hand image may be obtained.

[0056] Step 203: input the corrected hand image into a preset key point detection model to obtain the hand key point coordinates of the target user, and determine the hand key point coordinates and the connecting lines between adjacent hand key point coordinates as the hand feature information of the target user.

[0057] Among them, the preset key point detection model is a key point detection model that can be pre-set according to actual conditions for detecting the key points of the hand image after correction. There can be multiple preset key point detection models. As a possible implementation method, MediaPipe can be used to complete real-time detection and tracking of hand key points. MediaPipe is a multimedia machine learning model application framework developed and open sourced by Google Research, which can complete face recognition, iris detection, body tracking, etc.

[0058] For the embodiments of the present disclosure, the corrected hand image can be input into the MediaPipe model, and the hand key point coordinates and the connecting lines between adjacent hand key point coordinates can be obtained, and the hand key point coordinates and the connecting lines are determined as hand feature information, wherein the hand key point coordinates can be hand joint point coordinates, and the connecting lines can be joint connections, and no specific limitation is made to this; the hand feature information can be features such as hand joint point coordinates, joint connections, and no specific limitation is made to this.

[0059] Step 204: Use a convolutional neural network to perform feature processing on the hand feature information to determine the fingertip position of the target user.

[0060] For the implementation process of the embodiment of the present disclosure, please refer to the relevant description in step 103 of the embodiment, which will not be repeated here.

[0061] Step 205: determine the information recognition area corresponding to the fingertip position, and process the information to be scanned in the information recognition area according to the corresponding scanning function, where the scanning function includes a word scanning translation function and / or a question scanning answering function.

[0062] For the disclosed embodiment, before determining the information recognition area corresponding to the fingertip position and processing the information to be scanned in the information recognition area according to the corresponding scanning function, the scanning function corresponding to the information recognition area can be determined first. Accordingly, as a possible implementation method, the information to be scanned in the information recognition area can be compared with the preset information for information similarity. If the information similarity is greater than the preset similarity threshold, the preset scanning function corresponding to the preset information is determined as the scanning function corresponding to the information recognition area, wherein the preset similarity threshold is a value that can be preset according to actual conditions (such as 80%), and there is no specific limitation on this; the information to be scanned is the information to be scanned in the information recognition area, and different information recognition areas correspond to different information to be scanned. The preset information is the information preset for similarity comparison with the information to be scanned, such as preset English, titles, words, etc., and there is no specific limitation on this. Different preset information corresponds to different preset scanning functions, and the preset scanning function is a preset scanning function corresponding to the preset information, such as a preset scanning word translation function, scanning question answering function, etc.

[0063] For the embodiment of the present disclosure, as another possible implementation method, the information features of the information in the information identification area can be extracted, and the information features can be input into the trained scanning function model to obtain the scanning function corresponding to the information identification area. Among them, the information features can be the information content in the information identification area, and the scanning function model can output the scanning function corresponding to the information features based on the information features in the information identification area. It can be understood that the scanning function model can be any computing model that can realize information content recognition, such as a neural network model, a deep learning model, etc., which is not limited in the present disclosure. Among them, the training method of the scanning function model, the specific implementation process can be described as: extracting the historical information features of the user in the historical time period, and the historical scanning function under the historical information features; using the historical information features as the input features of the scanning function model, using the historical scanning functions as the training labels of the scanning function model, iteratively training the scanning function model until the loss function of the scanning function model is less than the preset loss function threshold, and determining that the scanning function model training is completed, wherein the preset loss function threshold is a value that can be set according to the actual application scenario, and is not specifically limited here. The closer the setting value of the preset loss function threshold is to 0, the higher the recognition accuracy of the scanning function model.

[0064] For the embodiments of the present disclosure, when the information to be scanned in the information identification area is processed according to the corresponding scanning function, taking the scanning function including the word scanning and translation function and / or the question scanning and answering function as an example, the specific implementation process can be described as: performing word scanning and translation processing on the information to be scanned in the information identification area according to the word scanning and translation function; and / or performing question scanning and answering processing on the information to be scanned in the information identification area according to the question scanning and answering function.

[0065] In summary, according to an information processing method, device and electronic device provided by the present disclosure, compared with the current prior art, the present disclosure can obtain a hand image of a target user captured by a camera; perform image correction on the hand image, and obtain the hand feature information of the target user by performing feature recognition on the corrected hand image; perform feature processing on the hand feature information using a convolutional neural network to determine the fingertip position of the target user; determine the information recognition area corresponding to the fingertip position, and process the information to be scanned in the information recognition area according to the corresponding scanning function, and the scanning function includes a word scanning translation function and / or a question scanning and answering function. By adopting the above technical solution, after determining the hand feature information by performing feature recognition on the hand image, the information to be scanned can be automatically processed based on the hand feature information, which can improve the information input efficiency under the corresponding scanning function, thereby improving the user experience.

[0066] Based on the above Figure 1 and Figure 2 The specific implementation of the method shown in this embodiment provides an information processing device, such as Figure 3 As shown, the device includes: an acquisition module 31, a correction module 32, a processing module 33, and a determination module 34;

[0067] An acquisition module 31 may be used to acquire a hand image of a target user captured by a camera;

[0068] The correction module 32 may be used to perform image correction on the hand image and obtain hand feature information of the target user by performing feature recognition on the corrected hand image;

[0069] The processing module 33 may be used to perform feature processing on hand feature information using a convolutional neural network to determine the fingertip position of the target user;

[0070] The determination module 34 can be used to determine the information recognition area corresponding to the fingertip position, and process the information to be scanned in the information recognition area according to the corresponding scanning function. The scanning function includes a word scanning translation function and / or a question scanning and answering function.

[0071] In a specific application scenario, the correction module 32 can be used to calibrate the hand image through a preset calibration device to obtain the internal parameters and distortion parameters corresponding to the camera; remap the hand image based on the internal parameters, and reconstruct the grayscale of the remapped hand image based on the distortion parameters to obtain a corrected image corresponding to the hand image.

[0072] In a specific application scenario, the correction module 32 can be used to input the corrected hand image into a preset key point detection model to obtain the hand key point coordinates of the target user; the hand key point coordinates and the connecting lines between adjacent hand key point coordinates are determined as the hand feature information of the target user.

[0073] In a specific application scenario, the processing module 33 can be used to extract a first hand feature from the hand feature information using a deep network convolution layer of a convolutional neural network, and to extract a second hand feature from the hand feature information using a shallow network convolution layer of a convolutional neural network, wherein the first hand feature is all feature points in the fingertip area of ​​the hand, and the second hand feature is the texture edge of the fingertip area of ​​the hand; the first hand feature and the second hand feature are feature fused, and the result of the feature fusion is feature classified, and the fingertip position of the target user is extracted from the feature classification result.

[0074] In specific application scenarios, such as Figure 4 As shown, the device further includes: a comparison module 35, an extraction module 36;

[0075] In a specific application scenario, the comparison module 35 can be used to compare the information to be scanned in the information identification area with the preset information for information similarity. If the information similarity is greater than the preset similarity threshold, the preset scanning function corresponding to the preset information is determined as the scanning function corresponding to the information identification area, wherein the information to be scanned is the information to be scanned in the information identification area, different information identification areas correspond to different information to be scanned, and the preset information is pre-set information for similarity comparison with the information to be scanned, and different preset information corresponds to different preset scanning functions; or,

[0076] In a specific application scenario, the extraction module 36 can be used to extract information features of information in the information recognition area, input the information features into the trained scanning function model, and obtain the scanning function corresponding to the information recognition area.

[0077] In a specific application scenario, the extraction module 36 can be used to extract the historical information features of the user within the historical time period, as well as the historical scanning function under the historical information features; using the historical information features as the input features of the scanning function model, and using the historical scanning function as the training label of the scanning function model, iteratively train the scanning function model until the loss function of the scanning function model is less than the preset loss function threshold, and it is determined that the scanning function model training is completed.

[0078] In a specific application scenario, the determination module 34 can be used to perform word scanning and translation processing on the information to be scanned in the information identification area according to the word scanning and translation function; and / or perform question scanning and answering processing on the information to be scanned in the information identification area according to the question scanning and answering function.

[0079] It should be noted that for other corresponding descriptions of the functional units involved in the information processing device provided in this embodiment, please refer to Figure 1 and Figure 2 The corresponding description in will not be repeated here.

[0080] Based on the above Figure 1 and Figure 2 The method shown in the embodiment also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned Figure 1 and Figure 2 The method shown.

[0081] Based on this understanding, the technical solution of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, USB flash drive, mobile hard disk, etc.), including a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods of various implementation scenarios of the present application.

[0082] Based on the above Figure 1 and Figure 2 The method shown, and Figure 3 and Figure 4 In order to achieve the above-mentioned purpose, the embodiment of the present application further provides an electronic device which can be configured on the vehicle (such as an electric vehicle) side, and the device includes a storage medium and a processor; the storage medium is used to store a computer program; the processor is used to execute the computer program to achieve the above-mentioned Figure 1 and Figure 2 The method shown.

[0083] Optionally, the above-mentioned physical device may also include a user interface, a network interface, a camera, a radio frequency (RF) circuit, a sensor, an audio circuit, a WI-FI module, etc. The user interface may include a display, an input unit such as a keyboard, etc., and the optional user interface may also include a USB interface, a card reader interface, etc. The network interface may optionally include a standard wired interface, a wireless interface (such as a WI-FI interface), etc.

[0084] Those skilled in the art will appreciate that the above-mentioned physical device structure provided in this embodiment does not constitute a limitation on the physical device, and may include more or fewer components, or a combination of certain components, or different arrangements of components.

[0085] The storage medium may also include an operating system and a network communication module. The operating system is a program that manages the hardware and software resources of the above-mentioned physical device, and supports the operation of the information processing program and other software and / or programs. The network communication module is used to realize the communication between the components inside the storage medium, and the communication with other hardware and software in the information processing physical device.

[0086] Through the description of the above implementation methods, those skilled in the art can clearly understand that the present application can be implemented by means of software plus the necessary general hardware platform, or by hardware. By applying the scheme of this embodiment, compared with the current prior art, the present disclosure can obtain the hand image of the target user captured by the camera; perform image correction on the hand image, and obtain the hand feature information of the target user by performing feature recognition on the corrected hand image; perform feature processing on the hand feature information using a convolutional neural network to determine the fingertip position of the target user; determine the information recognition area corresponding to the fingertip position, and process the information to be scanned in the information recognition area according to the corresponding scanning function, and the scanning function includes a word scanning translation function and / or a question scanning and answering function. By adopting the above technical scheme, after performing feature recognition on the hand image and determining the hand feature information, the information to be scanned can be automatically processed based on the hand feature information, which can improve the information input efficiency under the corresponding scanning function, thereby improving the user experience.

[0087] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the term "comprising" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprising a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0088] The above is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to these embodiments herein, but will conform to the widest scope consistent with the principles and novel features applied for herein.

Claims

1. An information processing method, characterized in that: The method comprises: Obtain a hand image of the target user captured by the camera; Performing image correction on the hand image, and performing feature recognition on the corrected hand image to obtain hand feature information of the target user; Using a convolutional neural network to perform feature processing on the hand feature information to determine the fingertip position of the target user; Determine the information recognition area corresponding to the fingertip position, and process the information to be scanned in the information recognition area according to the corresponding scanning function, wherein the scanning function includes a word scanning translation function and / or a question scanning answering function.

2. The method according to claim 1, characterized in that: The performing image correction on the hand image comprises: Calibrate the hand image by using a preset calibration device to obtain internal parameters and distortion parameters corresponding to the camera; The hand image is remapped based on the internal parameters, and the grayscale of the remapped hand image is reconstructed based on the distortion parameters to obtain a corrected image corresponding to the hand image.

3. The method according to claim 1, characterized in that The obtaining hand feature information of the target user by performing feature recognition on the corrected hand image includes: Inputting the corrected hand image into a preset key point detection model to obtain the coordinates of the key points of the hand of the target user; The hand key point coordinates and the connecting lines between adjacent hand key point coordinates are determined as the hand feature information of the target user.

4. The method according to claim 1, characterized in that: The process of performing feature processing on the hand feature information by using a convolutional neural network to determine the fingertip position of the target user includes: Extracting a first hand feature from the hand feature information using a deep network convolution layer of the convolutional neural network, and extracting a second hand feature from the hand feature information using a shallow network convolution layer of the convolutional neural network, wherein the first hand feature is all feature points of a hand fingertip region, and the second hand feature is a texture edge of a hand fingertip region; The first hand feature and the second hand feature are subjected to feature fusion, and a result of the feature fusion is subjected to feature classification, and the fingertip position of the target user is extracted from the result of the feature classification.

5. The method according to claim 1, characterized in that Before determining the information recognition area corresponding to the fingertip position and processing the information to be scanned in the information recognition area according to the corresponding scanning function, the method further includes: Compare the information to be scanned in the information identification area with the preset information for information similarity; if the information similarity is greater than a preset similarity threshold, determine the preset scanning function corresponding to the preset information as the scanning function corresponding to the information identification area, wherein the information to be scanned is the information to be scanned in the information identification area, different information identification areas correspond to different information to be scanned, and the preset information is pre-set information for performing similarity comparison with the information to be scanned, and different preset information corresponds to different preset scanning functions; or The information features of the information in the information recognition area are extracted, and the information features are input into the trained scanning function model to obtain the scanning function corresponding to the information recognition area.

6. The method according to claim 5, characterized in that The training method of the scanning function model comprises: Extracting historical information features of users within a historical time period, and historical scanning functions under the historical information features; The historical information features are used as input features of the scanning function model, and the historical scanning functions are used as training labels of the scanning function model. The scanning function model is iteratively trained until the loss function of the scanning function model is less than a preset loss function threshold, and it is determined that the training of the scanning function model is completed.

7. The method according to claim 5, characterized in that The processing of the information to be scanned in the information identification area according to the corresponding scanning function includes: Performing word scanning and translation processing on the information to be scanned in the information recognition area according to the word scanning and translation function; and / or The information to be scanned in the information recognition area is scanned and answered according to the scan question and answer function.

8. An information processing device, characterized in that: include: An acquisition module, used to acquire a hand image of a target user captured by a camera; A correction module, used to perform image correction on the hand image, and obtain hand feature information of the target user by performing feature recognition on the corrected hand image; A processing module, used to perform feature processing on the hand feature information using a convolutional neural network to determine the fingertip position of the target user; A determination module is used to determine the information recognition area corresponding to the fingertip position, and process the information to be scanned in the information recognition area according to the corresponding scanning function, wherein the scanning function includes a word scanning translation function and / or a question scanning answering function.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. An electronic device comprising a storage medium, a processor, and a computer program stored in the storage medium and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

Images