Method and device for identifying injection attack behavior

Abstract

The embodiment of the invention provides an injection attack behavior identification method and device, and the method comprises the steps: obtaining first data corresponding to an image collection process of a mobile terminal for a first object, the first data comprises sensor data, the sensor data corresponds to a sensor in the mobile terminal, and the sensor data corresponds to the sensor in the mobile terminal; the sensor comprises a first sensor for sensing the movement of the mobile terminal; image data corresponding to the image acquisition process are obtained, the image data are obtained through recognition of a first image corresponding to the image acquisition process, and the image data are used for indicating movement of the mobile terminal; and determining whether the image acquisition process relates to the injection attack behavior or not by using the first data and the image data, so as to provide a new injection attack behavior identification method.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method and device for identifying injection attack behavior. Background Art

[0002] The eKYC (electronic Know Your Customer) system is a digital identity authentication solution based on artificial intelligence technology. Institutions in the fields of financial services, banking, insurance, telecommunications and e-commerce often use the eKYC system to authenticate their users.

[0003] At present, in the process of authenticating its users, the eKYC system generally calls the camera of the mobile terminal to capture the user's face and / or ID card, and then uses the captured face and / or ID card images (or videos, i.e., image sequences) to authenticate the user. However, with the continuous development of AIGC (Artificial Intelligence Generated Content) technology, there are cases where the black industry uses AIGC technology to generate images (or videos) of faces and / or ID cards to carry out injection attacks on the eKYC system. During this injection attack, the black industry exploits the vulnerabilities of the mobile terminal to replace the real images (or videos) captured by the camera of the mobile terminal with the generated images (or videos) of faces and / or ID cards, and injects them into the image (or video) verification unit of the eKYC system.

[0004] At present, in identity authentication systems such as eKYC systems, the defense process of the above injection attack is only carried out from the image (or video) itself. However, due to the continuous development of AIGC technology, its forged images (or videos) are becoming more and more realistic, making it increasingly difficult to defend only from the image (or video) itself. In order to ensure the privacy information and / or property security of customers of various institutions, it is crucial to provide a new method for identifying injection attack behaviors for defense. Summary of the invention

[0005] One or more embodiments of the present specification provide a method and device for identifying injection attack behaviors, so as to realize the identification of injection attack behaviors.

[0006] According to a first aspect, a method for identifying injection attack behavior is provided, comprising:

[0007] Acquire first data corresponding to an image acquisition process of a mobile terminal for a first object, wherein the first data includes sensor data, the sensor data corresponds to a sensor in the mobile terminal, and the sensor includes a first sensor for sensing a motion of the mobile terminal;

[0008] Acquire image data corresponding to the image acquisition process, wherein the image data is obtained from the first image recognition corresponding to the image acquisition process, and the image data is used to indicate the movement of the mobile terminal;

[0009] Using the first data and the image data, it is determined whether the image acquisition process involves injection attack behavior.

[0010] According to a second aspect, a device for identifying injection attack behavior is provided, comprising:

[0011] A first acquisition module is configured to acquire first data corresponding to an image acquisition process of a mobile terminal for a first object, wherein the first data includes sensor data, the sensor data corresponds to a sensor in the mobile terminal, and the sensor includes a first sensor for sensing a motion of the mobile terminal;

[0012] A second acquisition module is configured to acquire image data corresponding to the image acquisition process, wherein the image data is obtained from the first image recognition corresponding to the image acquisition process, and the image data is used to indicate the movement of the mobile terminal;

[0013] The first determination module is configured to use the first data and the image data to determine whether the image acquisition process involves injection attack behavior.

[0014] According to a third aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed in a computer, the computer is caused to execute the method described in the first aspect.

[0015] According to a fourth aspect, a computing device is provided, comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method described in the first aspect is implemented.

[0016] According to the injection attack behavior identification method and device provided in the embodiments of this specification, first data corresponding to the image acquisition process of the mobile terminal for the first object is obtained, wherein the first data includes sensor data, and the sensor data corresponds to a sensor in the mobile terminal, and the sensor includes a first sensor for sensing the movement of the mobile terminal; image data corresponding to the image acquisition process is obtained, wherein the image data is obtained from the first image recognition corresponding to the image acquisition process, and the image data is used to indicate the movement of the mobile terminal; and the first data and the image data are used to determine whether the image acquisition process involves injection attack behavior.

[0017] In the above process, a new method for identifying injection attack behavior is provided. Considering that in a real image acquisition process (i.e., an image acquisition process that does not involve injection attack behavior), the movement of the mobile terminal in the image acquisition process is associated with the movement of the first object in the image sequence acquired during the image acquisition process. For example, the mobile terminal moves to the right relative to the first object, and accordingly, the image sequence will indicate that the first object (and / or the background therein) moves to the left relative to the mobile terminal. For another example, the mobile terminal approaches the first object, and accordingly, the image sequence will indicate that the first object (and / or the background therein) becomes larger, that is, the image sequence can also indicate the movement of the mobile terminal. In view of this, in this method, by identifying and determining the association between the first data corresponding to the image acquisition process, i.e., the sensor data (which can indicate the movement of the mobile terminal during the image acquisition process), and the image data corresponding to the image acquisition process (which can also indicate the movement of the mobile terminal), for example, determining whether the movement of the mobile terminal indicated by the two is consistent, it is determined whether the image acquisition process involves injection attack behavior, thereby realizing the identification of injection attack behavior. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 A schematic diagram of an implementation framework of an embodiment disclosed in this specification;

[0020] Figure 2 A flow chart of a method for identifying injection attack behavior provided by an embodiment;

[0021] Figure 3 A schematic diagram of the identification process of injection attack behavior provided by the embodiment;

[0022] Figure 4 A schematic diagram of a process for determining image data provided by an embodiment;

[0023] Figure 5 A schematic diagram of an image acquisition situation provided in an embodiment;

[0024] Fig. 6A A schematic diagram of a flow chart for determining whether an image acquisition process involves injection attack behavior provided by an embodiment;

[0025] Figure 6BA schematic diagram of another flow chart for determining whether an image acquisition process involves injection attack behavior provided by an embodiment;

[0026] Figure 7 A schematic diagram of the identification process of injection attack behavior provided by the embodiment;

[0027] Figure 8 A schematic block diagram of a device for identifying injection attack behavior provided in an embodiment. DETAILED DESCRIPTION

[0028] The technical solutions of the embodiments of this specification will be described in detail below with reference to the accompanying drawings.

[0029] The embodiments of this specification disclose a method and device for identifying injection attack behaviors. The application scenarios and technical concepts of the method are first introduced as follows:

[0030] As mentioned above, currently, in identity authentication systems such as eKYC systems, the defense against injection attacks is only carried out from the image (or video) itself. However, due to the continuous development of AIGC technology, its forged images (or videos) are becoming more and more realistic, making it increasingly difficult to defend only from the image (or video) itself. In order to ensure the security of customer information / or property of various institutions, it is crucial to provide a new method for identifying injection attack behaviors for defense.

[0031] In view of this, the inventor proposes a method for identifying injection attack behavior. Figure 1 FIG. 1 is a schematic diagram of an implementation scenario according to an embodiment disclosed in this specification. In this implementation scenario, the identification process of injection attack behavior is executed on a mobile terminal that performs image acquisition. Figure 1 As shown, the mobile terminal may be provided with an image acquisition device and a sensor, wherein the sensor may include a first sensor for sensing the movement of the mobile terminal, and may also include a second sensor for sensing the ambient light condition of the environment where the mobile terminal is located.

[0032] Exemplarily, the first sensor may include but is not limited to: an accelerometer, a gyroscope, a magnetometer, etc., and the accelerometer may include a linear acceleration sensor and / or an angular acceleration sensor; the second sensor may include but is not limited to: a photodiode, a photoresistive diode, and other sensors that can sense ambient light.

[0033] The mobile terminal may be installed with a target program, which is run by a processor of the mobile terminal. The target program is run to obtain data corresponding to the image acquisition process, wherein the data may include sensor data corresponding to the sensor in the mobile terminal and image data recognized from the image corresponding to the image acquisition process; exemplarily, the target program may also be run to identify whether the image acquisition process involves injection attack behavior based on the acquired data corresponding to the image acquisition process. Exemplarily, the target program may exist in the form of an SDK (Software Development Kit). The target program may exist in the form of a functional plug-in of an identity authentication system.

[0034] Exemplarily, when a user needs to perform identity authentication, a mobile terminal can be used to collect images, and the collected images (sequences) can be used for identity authentication. In the embodiments of this specification, in order to ensure the privacy information and / or property security of users, before using images (sequences) for identity authentication, it is necessary to identify the authenticity of the images (sequences), that is, to identify whether the image acquisition process involves injection attacks, so as to avoid the black industry using forged images (sequences) to pass identity authentication, causing the privacy information and / or property of the corresponding user (the user corresponding to the forged images of the black industry) to be lost.

[0035] Specifically, when the user triggers the identity authentication function (for example, triggering the image acquisition button), the target program of the mobile terminal is triggered. Accordingly, the target program of the mobile terminal starts the image acquisition device of the mobile terminal in response to the user's trigger, so as to use the image acquisition device to acquire images of the first object. Subsequently, the target program acquires the first data corresponding to the image acquisition process for the first object, wherein the first data includes sensor data, the sensor data corresponds to the sensor in the mobile terminal, and the sensor includes a first sensor for sensing the movement of the mobile terminal; and acquires the image data corresponding to the aforementioned image acquisition process, wherein the image data is obtained from the first image recognition corresponding to the image acquisition process, and the image data is used to indicate the movement of the mobile terminal. Afterwards, the target program uses the first data and the image data to determine whether the image acquisition process involves injection attack behavior.

[0036] In the above process, a new method for identifying injection attack behavior is provided. Considering that in a real image acquisition process (i.e., an image acquisition process that does not involve injection attack behavior), the movement of the mobile terminal in the image acquisition process is associated with the movement of the first object in the image sequence acquired during the image acquisition process. For example, the mobile terminal moves to the right relative to the first object, and accordingly, the image sequence will represent that the first object (and / or the background in the image) moves to the left relative to the mobile terminal. For another example, the mobile terminal approaches the first object, and accordingly, the image sequence will represent that the first object (and / or the background in the image) becomes larger, that is, the image sequence can also indicate the movement of the mobile terminal. In view of this, in the implementation method provided in this specification, by identifying and determining the association between the first data corresponding to the image acquisition process, i.e., the sensor data (which can represent the movement of the mobile terminal during the image acquisition process), and the image data corresponding to the image acquisition process (which can also indicate the movement of the mobile terminal), for example, determining whether the movement of the mobile terminal indicated by the two is consistent, it is determined whether the image acquisition process involves injection attack behavior, thereby realizing the identification of injection attack behavior.

[0037] Moreover, in actual scenarios, the sensor data in the image acquisition process is often difficult to forge. Furthermore, it is relatively more difficult to forge sensor data and image data that have a consistent relationship in the same image acquisition process, that is, matching. In view of this, the above process can better realize the accurate detection of whether the image acquisition process involves injection attack behavior, better improve the security of the identity authentication system, and thus improve the privacy information / or property security of users.

[0038] The following is a detailed description of the method for identifying injection attack behaviors provided in this specification in conjunction with specific embodiments.

[0039] Figure 2 A flow chart of a method for identifying injection attack behavior in one embodiment of the present specification is shown. The method is executed by an electronic device, and the electronic device can be implemented by any device, equipment, platform, device cluster, etc. with computing and processing capabilities. In some examples, the electronic device can be a mobile terminal for capturing images of a first object, or a server that can communicate with a mobile terminal for capturing images of a first object. Exemplarily, the mobile terminal can be, but is not limited to, terminal devices such as mobile phones and tablet computers.

[0040] In the process of identifying injection attack behaviors, Figure 2 As shown, the method includes the following steps S210-S230:

[0041] In step S210, first data corresponding to an image acquisition process of a mobile terminal for a first object is acquired, wherein the first data includes sensor data corresponding to a sensor in the mobile terminal, and the sensor is used to sense movement of the mobile terminal.

[0042] It is understandable that in the current personnel identity verification system (such as the eKYC system), the user's identity is generally verified by identifying the card and / or the face. In some examples of this specification, the first object may be, but is not limited to, a card and / or a face. The card may include, but is not limited to, a bank card, various types of identity documents, etc. The various types of identity documents may include, but are not limited to, an identity card, a driver's license, and a passport.

[0043] The following takes the electronic device as a mobile terminal that performs image acquisition for the first object as an example to introduce the identification method of injection attack behavior provided by this embodiment. In the case where the electronic device is a server that can communicate with the mobile terminal that performs image acquisition for the first object, its implementation process is similar to the implementation process when the electronic device is a mobile terminal. Its implementation process can refer to the implementation process when the electronic device is a mobile terminal, and it will not be repeated here. Among them, in the case where the electronic device is a server, the mobile terminal needs to send the acquired first data and image data to the server through the network, so that the server can perform the action of using the first data and image data to determine whether the image acquisition device involves injection attack behavior; or, the mobile terminal sends the acquired first data and the image sequence corresponding to the image acquisition process to the server, so that the server obtains image data from the image sequence, and then uses the first data and image data to determine whether the image acquisition device involves injection attack behavior. Action, which is also possible.

[0044] In the case where the electronic device is a mobile terminal for collecting images of the first object, the mobile terminal is installed with a target program, such as Figure 1 As shown, the target program runs on the processor of the mobile terminal to execute the identification process of injection attack behavior. Figure 1 As shown, the mobile terminal is also provided with an image acquisition device (such as a camera, etc.) for being called by the target program to perform image acquisition, and the mobile terminal is also provided with multiple types of sensors. The multiple types of sensors include, for example: multiple types of first sensors for sensing the movement of the mobile terminal. Exemplarily, the first sensor may include, for example, but is not limited to: an acceleration sensor, a gyroscope, and a magnetometer, etc., and the acceleration sensor may include a linear acceleration sensor and / or an angular acceleration sensor.

[0045] In some other examples, the multiple types of sensors may also include: a second sensor for sensing the ambient light of the environment in which the mobile terminal is located. Subsequently, the relationship between the ambient light conditions corresponding to the second sensor corresponding to the image acquisition process and the ambient light conditions indicated in the image data corresponding to the image acquisition process may be used to assist in determining whether the image acquisition process involves injection attack behavior, so as to identify whether the image acquisition process involves injection attack behavior from multiple angles and better improve the recognition accuracy. The second sensor may include, for example, but is not limited to, the following types of sensors: photodiodes, photoresist diodes, and other sensors that can sense ambient light conditions (e.g., strength).

[0046] The specific identification process of injection attack behavior is introduced below. The identification process of injection attack behavior can include two stages, namely, the data acquisition stage corresponding to the image acquisition process and the specific identification stage. The data acquisition stage corresponding to the image acquisition process is introduced below.

[0047] In some exemplary scenarios, when a user needs to handle certain specific services online, in order to ensure the privacy information and property security of users related to the specific services, the user's identity must be authenticated first. Only after the user's identity authentication is passed can the user continue to handle certain specific services. In order to ensure the privacy information and property security of the user, it is necessary to verify the authenticity of the image (sequence) for identity authentication, that is, to verify and identify whether the image acquisition process corresponding to the identity authentication involves injection attack behavior. Accordingly, the user triggers the identity authentication function (for example, triggers the image acquisition button), and the target program of the mobile terminal is triggered to obtain data corresponding to the image acquisition process of the mobile terminal for the first object.

[0048] Specifically, after the target program is triggered, the target program can trigger the image acquisition device of the mobile terminal to start the image acquisition device, and then perform image acquisition through the image acquisition device. During the image acquisition process, the target program can provide prompt information to the user to instruct the user to use a specific acquisition method to acquire images of the first object. Accordingly, in the real image acquisition process (i.e., the image acquisition process that does not involve injection attack behavior), the user will control the mobile terminal to move based on the prompt information of the target program, so that the image acquisition device of the mobile terminal can acquire images that meet the specific acquisition method for the first object. In the real image acquisition process, the movement of the mobile terminal is controlled by the user based on the prompt information of the target program (i.e., the indicated specific acquisition method). That is to say, in the real image acquisition process, the user controls the movement of the mobile terminal based on the prompt information of the target program (the user's behavior is the user's acquisition behavior) to use the mobile terminal to acquire images for the first object.

[0049] During the actual image acquisition process, the target program may obtain an image sequence for the first object that is actually acquired by the image acquisition device.

[0050] In the process of image acquisition involving injection attack behavior, the mobile terminal is often stationary in a certain place, that is, it will not be moved by the user based on the prompt information of the target program (that is, the specific acquisition method indicated) to perform image acquisition. At this time, the black industry (for example, a program installed on the mobile terminal) will replace the image sequence actually acquired by the image acquisition device with its forged image sequence, and then transmit the forged image sequence to the target program to achieve the injection attack. At this time, the target program obtains the image sequence forged by the black industry.

[0051] In the above-mentioned image acquisition process, the sensor in the mobile terminal will sense the movement of the mobile terminal and the ambient light of the environment in which it is located, and measure and obtain corresponding data. Specifically, in the above-mentioned image acquisition process, the first sensor in the mobile terminal can sense the movement of the mobile terminal in the process, and measure and obtain corresponding data (hereinafter referred to as the first measurement data) for the movement of the mobile terminal in the process, and the first measurement data can characterize the movement of the mobile terminal in the process. In addition, the second sensor in the mobile terminal can sense the ambient light conditions of the environment in which the mobile terminal is located during the process, and measure and obtain corresponding data (hereinafter referred to as the second measurement data) for the ambient light of the environment in which the mobile terminal is located during the process, and the second measurement data can characterize the ambient light conditions of the environment in which the mobile terminal is located during the process. Afterwards, the first sensor (and the second sensor) can send the measurement data obtained by its measurement to the target program.

[0052] Subsequently, the target program may obtain data corresponding to the sensors (including the first sensor and the second sensor) in the mobile terminal.

[0053] In some possible examples, the sensor data in the first data may include: data corresponding to each first sensor and / or posture data of the mobile terminal determined based on the data corresponding to the first sensor, wherein the posture data may indicate the position and posture of the mobile terminal. Exemplarily, the posture of the mobile terminal may be represented by Euler angles, which include: the rotation angle of the x-axis in the three-dimensional space, i.e., the pitch angle pitch, the rotation angle of the z-axis in the three-dimensional space, i.e., the roll angle roll, and the rotation angle of the y-axis in the three-dimensional space, i.e., the yaw angle yaw. The first data may also include data corresponding to the second sensor, such as ambient light data. Specifically, the first data may include sensor data corresponding to multiple acquisition timestamps. The multiple acquisition timestamps all belong to the time range corresponding to the image acquisition process.

[0054] In some examples, the posture data may be determined by the target program based on data corresponding to the first sensor; in other examples, the posture data may also be obtained by the target program from the first program, wherein the first program may determine the corresponding posture data based on the data corresponding to the first sensor corresponding to the image acquisition process, and then provide it to the target program.

[0055] The following introduces the determination of posture data by taking the first sensor including a linear acceleration sensor and a gyroscope as an example. Among them, the linear acceleration sensor can measure the linear acceleration of the mobile terminal on the x, y, and z axes in the three-dimensional space, and the gyroscope can measure the rotation speed of the mobile terminal rotating in the x, y, and z axes. By integrating the data corresponding to the linear acceleration sensor and the gyroscope, the posture change of the mobile terminal during the image acquisition process relative to the initial state (the position and posture of the mobile terminal when the image acquisition device of the mobile terminal starts to acquire images) can be estimated, so that any relevant technology can be used to determine a series of posture data of the mobile terminal during the image acquisition process according to the estimated posture change, that is, the posture data corresponding to each acquisition timestamp.

[0056] Specifically, in some possible examples, when the first data is the real data corresponding to the real image acquisition process, that is, when the above image acquisition process does not involve injection attack behavior, the acquired sensor data may include data directly measured by multiple types of sensors (for example, including the first sensor and the second sensor) in the mobile terminal (that is, the aforementioned first measurement data) and / or the posture data of the mobile terminal determined based on the data measured by the first sensor; it may also include data directly measured by the second sensor (that is, the aforementioned second measurement data). At this time, this type of sensor data is consistent with the user's behavior of controlling the movement of the mobile terminal based on the prompt information of the target program (also referred to as the user's acquisition behavior), that is, it is consistent with the user's acquisition behavior corresponding to the specific acquisition method adopted by the user in the image acquisition process.

[0057] In some other possible examples, when the first data is injected data corresponding to the image acquisition process, that is, the above-mentioned image acquisition process involves an injection attack, the acquired sensor data may include: forged data corresponding to multiple types of sensors (for example, including a first sensor and a second sensor) (that is, data measured by a forged first sensor), and / or forged posture data of the mobile terminal determined based on the forged data corresponding to the first sensor; it may also include forged data corresponding to the second sensor (that is, data measured by a forged second sensor).

[0058] In some specific examples, when the above-mentioned image acquisition process involves injection attack behavior, after the sensor in the mobile terminal measures and obtains measurement data, in the process of transmitting the measured data to the target program, the black industry will intercept the measurement data obtained by the sensor in the mobile terminal, and replace the measurement data transmitted by the sensor in the mobile terminal to the target program with forged data corresponding to multiple types of sensors, thereby transmitting the forged data to the target program. In the latter implementation, the target program continues to determine the posture data of the mobile terminal based on the obtained forged data forged by the black industry to obtain forged sensor data; or, in another implementation, the data forged by the black industry includes the measurement data forged for the sensor in the mobile terminal and the corresponding posture data of the mobile terminal, and accordingly, the target program directly obtains the corresponding forged sensor data.

[0059] In some other examples, when the above-mentioned image acquisition process involves injection attack behavior, in another case, the sensor data obtained by the target program may also be the data actually measured by the sensor of the mobile terminal and / or the posture data obtained based on the data actually measured by the sensor of the mobile terminal. However, at this time, this type of sensor data may not conform to the user's behavior of controlling the movement of the mobile terminal based on the prompt information of the target program, that is, it does not conform to the user's acquisition behavior corresponding to the specific acquisition method adopted by the user in the image acquisition process. For example, in the image acquisition process involving injection attack behavior, the mobile terminal may be stationary in a certain place (that is, the user does not control the mobile terminal to move according to the prompt information of the target program, that is, the user's acquisition behavior). At this time, the data measured by the sensor in the mobile terminal may be 0, etc.

[0060] In some other examples, when the above-mentioned image acquisition process involves injection attack behavior, in another case, the target program cannot obtain the measurement data of the sensor. At this time, the target program can directly determine that the image acquisition process involves injection attack behavior and does not continue to execute subsequent processes.

[0061] Each time the target program acquires data corresponding to the sensor in the mobile terminal, it can simultaneously record the timestamp corresponding to the data corresponding to the sensor in the mobile terminal acquired each time. The timestamp can be the acquisition timestamp mentioned later. Accordingly, the first data can include sensor data corresponding to multiple acquisition timestamps in the image acquisition process. In some examples, the sensor data may be data corresponding to the period between the start and the end of the image acquisition process, or may be data corresponding to each time stamp of a preset duration in the period between the start and the end of the image acquisition process.

[0062] It is understandable that when a mobile terminal has multiple sensors, the frequency of data measurement between different sensors may be different. After the target program obtains the data corresponding to multiple sensors, it can use any data alignment method in the relevant technology to align the data corresponding to the multiple sensors.

[0063] In some implementations, the aforementioned first data may also include acquisition method data, which is used to instruct the user to adopt a specific acquisition method during the image acquisition process. It can be understood that the acquisition method data is used to instruct the user to adopt a specific acquisition method during the image acquisition process, which is generated by the target program and cannot be forged by the black market. Accordingly, in order to better improve the accuracy of identifying whether the image acquisition process involves an injection attack, the first data may also include acquisition method data. Specifically, the first data may include acquisition method data corresponding to multiple acquisition timestamps during the image acquisition process.

[0064] The following is an introduction to the process of obtaining acquisition method data during image acquisition:

[0065] In some examples, after the target program detects that the image acquisition device is started, it starts the image acquisition process for the first object. During the image acquisition process, the target program may first output a first prompt information to prompt the user to perform a frontal image acquisition for the first object (e.g., Figure 3 As shown), accordingly, in the actual image acquisition process, the user will control the movement of the mobile terminal based on the first prompt information, so that the image acquisition device of the mobile terminal can acquire the front image of the first object. After acquiring the front image of the first object, the target program records the end time of the front acquisition subprocess of the first object (hereinafter referred to as the first end time). Exemplarily, the first end time can be the time when the target program detects that the front image of the first object that meets the preset conditions has been acquired, or it can be the time when the user is detected to trigger the front acquisition end button.

[0066] Subsequently, the target program may generate acquisition mode data based on the start time and the first end time of the image acquisition device, and include it in the first data corresponding to the image acquisition process. Exemplarily, between the start time and the first end time of the image acquisition device (i.e., the front acquisition subprocess of the first object), the acquisition mode data includes data indicating the front acquisition mode. That is, the acquisition timestamps corresponding to the sensor data between the start time and the first end time of the image acquisition device include data indicating the front acquisition mode.

[0067] Among them, exemplarily, the aforementioned frontal acquisition may refer to a acquisition method in which the first object is not tilted, for example, the plane where the first object is located is parallel to the plane where the display screen of the mobile terminal is located. It can also be considered that the plane where the first object is located is parallel to the plane where the image acquisition device of the mobile terminal is located. The parallelism may refer to that the angle between the plane where the first object is located and the plane where the display screen of the mobile terminal is located is less than a preset angle threshold.

[0068] Considering that in the process of injection attack behavior (i.e., the image acquisition process involving injection attack behavior), the black industry may forge sensor data and inject the forged sensor data into the target program to deceive the target program, so that the target program thinks that the forged sensor data is the measurement data measured by the sensor of the mobile terminal during the image acquisition process (and / or the determined posture data of the mobile terminal); and considering that the acquisition method data is data indicating that the user adopts a specific acquisition method during the image acquisition process (i.e., the aforementioned prompt information), it is generated by the target program and cannot be forged by the black industry. In view of this, in order to increase the complexity of the sensor data corresponding to the image acquisition process forged by the black industry, so that it is impossible to forge sensor data that can meet the user's specific acquisition method, so as to better improve the recognition detection rate and accuracy of injection attack behavior, and improve the defense against injection attack behavior, the embodiments of this specification design a more random and complex acquisition method, and / or a random vibration acquisition method.

[0069] In some possible examples, the acquisition mode data may include: data indicating a frontal acquisition mode and / or data indicating a non-frontal acquisition mode, and the non-frontal acquisition mode includes: a random multi-tilt angle acquisition mode. Specifically, the random multi-tilt angle acquisition mode may include instructing the user to perform image acquisition with a specified tilt information for the first object (such as Figure 3 The designated tilt information is randomly generated.

[0070] Exemplarily, when the first object is a human face, the specified tilt information may include: information indicating that an image is captured for the left side of the face and / or that an image is captured for the right side of the face.

[0071] In another exemplary embodiment, when the first object is a card, the specified tilt information may include indicating the tilt direction and / or tilt angle of the card, and the tilt mode and / or tilt angle are randomly generated to ensure randomness, thereby better improving the difficulty and complexity of the black industry to forge sensor data that can meet the user's specific collection method. Exemplarily, the tilt direction includes, but is not limited to, at least one of the following: tilting upward, tilting downward, tilting to the left, and tilting to the right. In another exemplary embodiment, the tilt angle includes, but is not limited to, at least one of the following: 15 degrees, 30 degrees, 40 degrees, etc.

[0072] Accordingly, in some possible examples, during the image acquisition process for the first object, after the target program detects that the front acquisition subprocess of the first object has ended, the target program may continue to output the second prompt information to prompt the user to perform non-front acquisition for the first object, i.e., to perform image acquisition in a non-front acquisition manner. The second prompt information may include randomly generated information instructing the user to perform non-front acquisition of the first object with a certain tilt information.

[0073] Accordingly, in the actual image acquisition process, the user can control the movement of the mobile terminal based on the second prompt information, so that the image acquisition device of the mobile terminal can acquire a non-frontal image of the first object that meets the second prompt information.

[0074] Subsequently, a non-frontal image of the first object that satisfies the second prompt information is acquired, and the target program records the end time of the non-frontal acquisition subprocess of the first object (hereinafter referred to as the second end time). The second end time may be the time when the target program detects that the non-frontal image of the first object that meets the preset conditions is acquired, or the time when the user triggers the non-frontal acquisition end button. Subsequently, the target program may generate acquisition mode data based on the startup time of the aforementioned image acquisition device, the first end time, the second end time, and the tilt information included in the second prompt information, and include it in the first data corresponding to the image acquisition process.

[0075] Exemplarily, between the first end time and the second end time of the image acquisition device (i.e., the non-frontal acquisition subprocess of the first object), the acquisition mode data may include data indicating the non-frontal acquisition mode, and may also include the aforementioned specified tilt information. That is, each corresponding acquisition timestamp of sensor data between the first end time and the second end time of the image acquisition device has its corresponding acquisition mode data including data indicating the non-frontal acquisition mode and / or specified tilt information.

[0076] It can be understood that the non-frontal acquisition method is optional, that is, it can appear randomly, that is, in the image acquisition process for different first objects, the target program can randomly instruct the user whether to adopt the non-frontal acquisition method, so that the image acquisition device acquires the corresponding non-frontal image of the first object.

[0077] In some other possible examples, considering that the mobile terminal is set with a preset vibrator, when the preset vibrator vibrates, it will have a certain impact on the measurement data obtained by the sensor measurement of the mobile terminal. In order to better improve the difficulty of the black industry to forge sensor data corresponding to the image acquisition process, and improve the recognition detection rate and accuracy of injection attack behaviors, the target program can also be set as follows: during the image acquisition process, the mobile terminal (the target program in) randomly generates a first instruction, and the first instruction is used to instruct the preset vibrator in the mobile terminal to vibrate; accordingly, the aforementioned first data can also include: data indicating whether the mobile terminal vibrates. It can be understood that during the image acquisition process, the target program of the mobile terminal can randomly generate a first instruction to randomly instruct the preset vibrator in the mobile terminal to vibrate, so that the mobile terminal randomly vibrates (such as Figure 3 As shown), and the first instruction may carry information indicating the vibration duration, wherein the vibration duration may be randomly generated to indicate that a preset vibrator in the mobile terminal may vibrate for an indefinite period of time, so as to better improve the randomness and increase the complexity and difficulty of the sensor data corresponding to the black market counterfeit image acquisition process.

[0078] It can be understood that during the above image acquisition process, the sensor of the mobile terminal has been sensing the movement of the mobile terminal (and sensing the ambient light conditions of the environment in which the mobile terminal is located), and measuring and obtaining its measurement data (such as Figure 3 The “Sensor Measurement Data” shown is throughout the entire image acquisition process).

[0079] Correspondingly, in the above example, the first data includes sensor data corresponding to multiple acquisition timestamps in the image acquisition process, acquisition mode data, and data indicating whether the mobile terminal is vibrating.

[0080] After obtaining the first data corresponding to the image acquisition process, in step S220, image data corresponding to the image acquisition process is obtained, wherein the image data is obtained from the first image recognition corresponding to the image acquisition process, and the image data is used to indicate the movement of the mobile terminal.

[0081] In some possible examples, the image data includes each image data identified from the first images corresponding to the aforementioned multiple acquisition timestamps.

[0082] In some examples, the change in the posture and / or other attributes of the first object in a series of images (i.e., the first image corresponding to multiple acquisition timestamps) can indicate the relative change between the first object and the mobile terminal, and accordingly, can indicate the movement of the mobile terminal, etc. For example, if the image indicates that the first object moves to the left relative to the mobile terminal, it can indicate that the mobile terminal moves to the right; if the image indicates that the first object becomes larger (the image area where the first object is located accounts for a larger proportion in the image), it can indicate that the mobile terminal is close to the first object.

[0083] In some other examples, the motion trajectory corresponding to the background (including objects other than the first object) in a series of images (i.e., the first image corresponding to multiple acquisition timestamps) can also indicate the movement of the mobile terminal. Accordingly, the image data may include relevant feature data of the first object identified from the first image (i.e., subsequent object feature data), and / or relevant feature data corresponding to the background identified from the first image (i.e., subsequent background feature data).

[0084] It can be understood that the target program has a similar process for obtaining image data from any first image. The process of obtaining image data from any first image is described below by taking any first image as an example.

[0085] Some possible examples include Figure 4 As shown, in step S220, step 11 may be included:

[0086] In step 11, when it is detected that the first image contains the first object, object feature data corresponding to the first object is detected from the first image and included in the image data corresponding to the first image.

[0087] In some implementations, the first image corresponding to the image acquisition process may be an image actually acquired by an image acquisition device of the mobile terminal; or the first image corresponding to the image acquisition process may be a forged image injected by a black market.

[0088] In some exemplary scenarios, after the image acquisition device is triggered and started by the target program, the display screen of the mobile terminal may display an image preview page, in which an image acquisition frame may be displayed, such as Figure 5 As shown, the user is instructed to set the first object at a suitable position so that the first object can be completely displayed in the image acquisition frame. When the image acquisition device acquires an image, the image acquisition area can be larger than the area of ​​the image acquisition frame to facilitate the acquisition of image data. Figure 5 As shown, the image acquisition device can acquire the image corresponding to the first frame. Exemplarily, the image preview page can also display the aforementioned prompt information to prompt the user to acquire the image in a specific acquisition method.

[0089] Specifically, in a real image acquisition process (i.e., an image acquisition process that does not involve injection attack behaviors), after the image acquisition device is started, the target program can obtain an image from the image acquisition device in real time as the first image. In an image acquisition process involving injection attack behaviors, the black industry will intercept the image (sequence) transmitted by the image acquisition device to the target program, and replace the intercepted image transmitted by the image acquisition device to the target program with its forged image (sequence), and transmit it to the target program, and the target program obtains the image (sequence) and uses it as the first image.

[0090] Subsequently, the target program may detect the first image it acquires to detect whether the first image contains the first object. When it is detected that the first image contains the first object, the object feature data corresponding to the first object is detected from the first image and included in the image data corresponding to the first image.

[0091] Exemplarily, a pre-trained first target recognition network or a traditional image recognition algorithm may be used to detect the first image to detect whether the first image contains the first object. In some examples, the first target recognition network may be a neural network pre-trained based on sample images containing the first object and their corresponding labels, and the neural network may be a CNN (convolutional neural network).

[0092] In some possible implementations, the aforementioned process of detecting object feature data corresponding to the first object from the first image may include: after detecting the first image using a pre-trained first object recognition network, the first object recognition network may output first position information of the first object in the first image when determining that the first image contains the first object, and the first position information is the regional position of the first object in the first image recognized by the first object recognition network; subsequently, based on the first position information, the regional position information of the image area where the first object in the first image is located may be determined.

[0093] In one implementation, the first position information is used as the regional position information of the image region where the first object in the first image is located. In another implementation, the first position information has a relatively low accuracy. In order to obtain the regional position information of the first object with better accuracy and precision, after the target program determines the first position information, it cuts out the regional image corresponding to the first position information from the first image; then continues to detect the regional image, detects the regional position information of the image region where the first object is located from the regional image, and classifies the regional position information into the object feature data corresponding to the first object.

[0094] The pre-trained second target recognition network or the traditional image recognition algorithm can be used to detect the regional position information of the image region where the first object is located from the regional image. The second target recognition network can be a neural network pre-trained based on a sample image containing the first object and a label corresponding to the location region of the first object in the sample image, and the neural network can be a CNN (convolutional neural network).

[0095] Next, the target program of the mobile terminal inputs the first image or the aforementioned regional image captured from the first image into a pre-trained attribute recognition network to obtain attribute data corresponding to the first object, and then classifies the attribute data corresponding to the first object into the object feature data corresponding to the first object. The attribute data may include but is not limited to: Euler angles, clarity values ​​and brightness values ​​corresponding to the first object. The attribute recognition network may be a neural network pre-trained based on a sample image containing the first object and its corresponding label indicating the attribute data of the first object in the sample image. It can be a neural network of any structure, such as a convolutional neural network, a transformer-based network, etc.

[0096] After the target program obtains the regional position information of the image area where the first object in the first image is located, it can also calculate the proportion of the image area where the first object is located in the first image (referred to as the first proportion) based on the regional position information, and then determine the first estimated distance between the first object and the mobile terminal based on the first proportion, and attribute the first estimated distance to the object feature data corresponding to the first object. It can be understood that the larger the proportion of the image area where the first object is located in the first image, the smaller the first estimated distance between the first object and the mobile terminal. Exemplarily, the corresponding relationship between each proportion of the image area where the first object is located in its corresponding image and each estimated distance can be preset, so that the corresponding relationship can be adopted to determine the first estimated distance between the first object and the mobile terminal based on the first proportion.

[0097] Accordingly, in some possible examples, the object feature data includes at least one of the following types of data: regional position information of the image area where the first object is located in the first image, the Euler angle, clarity value and brightness value corresponding to the first object, and the first estimated distance between the first object and the mobile terminal, wherein the first estimated distance is determined based on the proportion of the image area where the first object is located in the first image, and the Euler angle, clarity value and brightness value corresponding to the first object are determined based on the first image and a pre-trained attribute recognition network. Among them, the larger the brightness value corresponding to the aforementioned first object, the stronger the ambient light in the environment where the mobile terminal is located.

[0098] It is understandable that the above is only an exemplary example of object feature data and does not constitute a limitation on the object feature data. Any feature data that can indicate the relative relationship between the first object and the mobile terminal and correspondingly indicate the movement of the mobile terminal can be identified from the image and included in the object feature data. For example, the object feature data may also include data indicating whether the first object contained in the first image is complete.

[0099] In some possible examples, when the first object is a card, when capturing an image of the first object, the first object may be held by the user for image capture, or placed by the user on a plane (such as a desktop, etc.) (i.e., not held) for image capture. When the first object is held by the user, i.e., the first object is in a handheld state, during the image capture process, the user may control the movement of the first object and the movement of the mobile terminal at the same time, so that the image capture device can capture an image that satisfies the prompt information output by the target program (i.e., satisfies the specific capture method adopted by the user).

[0100] In some other possible examples, when the first object is a human face, it may also occur that during the image acquisition process, the human face moves based on the prompt information of the target program by the user, and the user controls the movement of the mobile terminal so that the image acquisition device can acquire an image that meets the prompt information output by the target program (that is, meets the specific acquisition method adopted by the user).

[0101] In the above situation, if the movement of the mobile terminal is indicated based on the changes in the image of the face or the card in the handheld state (such as the changes in the Euler angles and regional position information in the posture data, i.e. the object feature data), the movement of the mobile terminal indicated by the image is not accurate enough, which may cause errors in the match between the movement of the mobile terminal indicated by the image data and the movement of the mobile terminal indicated by the sensor data actually measured by the sensor of the mobile terminal in the corresponding process.

[0102] In view of the above situation, in order to ensure the accuracy of identifying whether the image acquisition process involves injection attack behavior. In some possible examples, such as Figure 4 As shown, in step S220, steps 12-13 may also be included:

[0103] In step 12, when it is detected that the first image contains the first object, the image area where the first object is located is subtracted from the first image to obtain a background area image corresponding to the first image.

[0104] In step 13, background feature data is extracted from the background area image and included in the image data corresponding to the first image, wherein the background feature data is used to calculate the motion of the first image relative to its previous frame image.

[0105] In some possible examples, when the target program detects that the first image contains the first object, the target program may deduct the image area where the first object is located from the first image based on the regional position information of the image area where the first object is located in the first image identified above, to obtain a background area image corresponding to the first image. Thereafter, background feature data is extracted from the background area image as image data corresponding to the first image, wherein the background feature data is used to calculate the motion of the first image relative to its previous frame image.

[0106] In some other possible examples, when the target program detects that the first object contained in the first image is a card, it can continue to identify and detect whether the first object is in a handheld state or placed on a desktop. Afterwards, when it is identified that the first object (such as a card) is in a handheld state, or when it is detected that the first object contained in the first image is a human face, steps 12-13 are executed to reduce the computing consumption of the mobile terminal to a certain extent.

[0107] In some implementations, the target program can detect and identify whether the first object is in a handheld state or in a state of being placed on a plane (or called a non-handheld state) by using a pre-trained state recognition network and the first image or a region image captured from the first image. The state recognition network can be a network pre-trained based on a specified sample image and its corresponding label, and the network can be, for example, a convolutional neural network, a transformer-based network, etc.

[0108] Among them, the designated sample image includes an image containing the first object in a handheld state and an image containing the first object in a non-handheld state; for the image containing the first object in a handheld state, the corresponding label is a label indicating that the first object is in a handheld state; for the image containing the first object in a non-handheld state, the corresponding label is a label indicating that the first object is in a non-handheld state.

[0109] Wherein, the aforementioned process of extracting background feature data from the background area image may include: extracting background feature data from the background area image using a specified feature extraction algorithm. Exemplarily, the specified feature extraction algorithm may be, but is not limited to, feature extraction algorithms such as SIFT (Scale-Invariant Feature Transform) algorithm, optical flow tracking algorithm, ORB (Oriented FAST and Rotated BRIEF) or speeded uprobust features (SURF). Exemplarily, the background feature data may include each feature point extracted based on the specified feature extraction algorithm. Exemplarily, the background feature data may also include a background brightness value, which is also possible. Wherein, the background brightness value may be determined by any image brightness determination algorithm in the related art, and the background brightness value may indicate the ambient light condition of the environment in which the mobile terminal is located to a certain extent, wherein the larger the background brightness value, the stronger the ambient light in the environment in which the mobile terminal is located.

[0110] In some possible examples, when the first image does not contain the first object, the value of the corresponding object feature data may be set to a specified value (e.g., 0). In some possible implementations, when the first image does not contain the first object, the brightness value in the object feature data may be set based on the brightness value of the entire first image, and the values ​​of other data in the object feature data except the brightness value may be set to a specified value.

[0111] In some other possible examples, such as Figure 4 As shown, in step S220, step 14 may also be included:

[0112] In step 14, when it is detected that the first image does not contain the first object, background feature data is extracted from the first image and included in the image data corresponding to the first image, wherein the background feature data is used to calculate the motion of the first image relative to its previous frame image.

[0113] In the above example, when the target program detects that the first image does not contain the first object, it can continue to extract background feature data from the first image and include it in the image data corresponding to the first image, so as to calculate the motion of the first image relative to its previous frame image based on the specified feature data to indicate the motion of the mobile terminal. The implementation principle of extracting background feature data from the first image is similar to the implementation principle of the aforementioned step 13, and its implementation method can refer to the implementation method of the aforementioned step 13, which will not be described in detail here.

[0114] After obtaining the first data and the image data, in step S230, the first data and the image data are used to determine whether the image acquisition process involves injection attack behavior.

[0115] Considering that in the actual image acquisition process, in the actual acquired image sequence (the first image corresponding to multiple acquisition timestamps) obtained by the target program from the image acquisition device, each image can represent that the posture of the background and / or the first object therein is changing, which can indicate the movement of the mobile terminal during the image acquisition process; and, the posture of the mobile terminal during the image acquisition process generally also changes, and accordingly, the data actually collected by the sensor of the mobile terminal (such as the first sensor) also changes, and the changes in the posture of the background and / or the first object represented by the image match the changes in the actual sensor data corresponding to the sensor of the mobile terminal during the image acquisition process.

[0116] In addition, during the actual image acquisition process, the images can also represent the changes in the environmental conditions (such as ambient light conditions) in which the mobile terminal is located, and the data actually collected by the sensor of the mobile terminal (such as the second sensor) can also represent the changes in the environmental conditions (such as ambient light conditions) in which the mobile terminal is located, and the changes represented by the two are matched.

[0117] In the image acquisition process involving injection attack behavior, in one implementation, the image sequence corresponding to the image acquisition process is forged by the black market, and the sensor data in the first data corresponding to the image acquisition process is obtained based on the data actually acquired by the sensor of the mobile terminal, which can indicate the actual movement of the mobile terminal (and the ambient light conditions of the actual environment in which the mobile terminal is located). At this time, the movement of the mobile terminal indicated by the image data corresponding to each image in the image sequence is less likely to match the movement of the mobile terminal indicated by the sensor data in the first data (and the ambient light conditions of the environment in which the mobile terminal is located indicated by the image data corresponding to each image in the image sequence are also less likely to match the ambient light conditions of the actual environment in which the mobile terminal is located indicated by the sensor data in the first data).

[0118] In another implementation of the image acquisition process involving injection attack behavior, the image sequence corresponding to the image acquisition process and the sensor data in the first data are both forged by the black market, and the possibility of matching the movement of the mobile terminal indicated by the two is low (and the possibility of matching the ambient light conditions of the mobile terminal's environment indicated by the image data corresponding to each image in the image sequence and the actual ambient light conditions of the mobile terminal's environment indicated by the sensor data in the first data is also low); further, the acquisition method data in the first data is generated by the target program itself, which cannot be forged, and accordingly, the movement of the mobile terminal indicated by the image data corresponding to each image in the image sequence and the movement of the mobile terminal indicated by the sensor data in the first data are less likely to match the acquisition method data in the first data.

[0119] In view of the above situation, the target program can use the first data and the image data to determine whether the image acquisition process involves injection attack behavior.

[0120] Specifically, after acquiring the first data and the image data, the target program can determine at least the motion conditions of the mobile terminal indicated by each of the first data and the image data. In some possible examples, the image data may include object feature data and / or background feature data identified from each of the first images corresponding to the multiple acquisition timestamps; the motion conditions of the mobile terminal can be determined based on the object feature data and / or background feature data in the image data corresponding to each of the first images.

[0121] Taking the object feature data as an example, based on the regional position information and Euler angles of the first object in the object feature data corresponding to the i-th first image, and the regional position information and Euler angles of the first object in the object feature data corresponding to the i-th first image, the posture data (hereinafter referred to as image posture data) corresponding to the i-th first image of the mobile terminal can be determined, that is, the image posture data corresponding to the acquisition timestamp t_i corresponding to the i-th first image. Among them, the image posture data of the above example may include a rotation angle and a displacement vector; the rotation angle is used to indicate the rotation angle (including the rotation angle of the Euler angle) of the mobile terminal at time t_i relative to the mobile terminal at time t_(i-1) in the x, y, and z axes of the three-dimensional space; the displacement vector is used to indicate the distance of displacement of the mobile terminal at time t_i relative to the mobile terminal at time t_(i-1) in the x, y, and z axes of the three-dimensional space.

[0122] Take background feature data as an example for explanation, wherein the background feature data in the object feature data corresponding to each first image includes each feature point extracted based on the specified feature extraction algorithm; accordingly, a brute-force matcher or a fast approximate nearest neighbor search based on FLANN (Fast Library for Approximate Nearest Neighbors) can be used to match multiple feature point pairs between the i-th first image and the i-1-th first image. Then, the RANSAC (random sample consensus) algorithm or other possible screening algorithms can be used to eliminate feature point pairs that may be abnormal. Then, based on the feature point pairs after screening, the essential matrix corresponding to the i-th first image and the i-1-th first image is calculated, and the image pose data of the i-th first image relative to the i-1-th first image is obtained by decomposing the essential matrix, that is, the image pose data corresponding to the i-th first image is obtained, that is, the image pose data corresponding to the acquisition timestamp t_i corresponding to the i-th first image is obtained.

[0123] It should be noted that when the value of i is 1, the value of each data item in the relative posture data of the 1st first image relative to the non-existent 0th first image can be recorded as 0.

[0124] The sensor data corresponding to each of the multiple acquisition timestamps in the first data may include posture data corresponding to each acquisition timestamp; then, whether the image acquisition process involves injection attack behavior can be determined by determining whether the movement trend of the mobile terminal indicated by the image posture data corresponding to the multiple acquisition timestamps matches the movement trend of the mobile terminal indicated by the posture data in the sensor data corresponding to the multiple acquisition timestamps.

[0125] Among them, if it is determined that the movement trend of the mobile terminal indicated by the image posture data corresponding to each acquisition timestamp matches the movement trend of the mobile terminal indicated by the posture data in the sensor data corresponding to multiple acquisition timestamps, then it is determined that the image acquisition process does not involve injection attack behavior; conversely, if it is determined that the movement trend of the mobile terminal indicated by the image posture data corresponding to each acquisition timestamp does not match the movement trend of the mobile terminal indicated by the posture data in the sensor data corresponding to multiple acquisition timestamps, then it is determined that the image acquisition process involves injection attack behavior.

[0126] Subsequently, in some examples, after determining that the image acquisition process does not involve injection attack behavior, the first image acquired during the image acquisition process can be used to authenticate the user. In other examples, after determining that the image acquisition process involves injection attack behavior, on the one hand, the user can be prompted to continue to re-acquire the image; on the other hand, in order to ensure the privacy information and / or property security of the user, the user's account can be locked, and the user can be notified of abnormalities in his or her related account through other contact methods set by the user to help the user avoid risks.

[0127] In some other examples, the ambient light conditions of the environment in which the mobile terminal is located, which are indicated by each of the first data and the image data, can also be determined. Specifically, the first brightness change value corresponding to each first image is determined based on the brightness value corresponding to the first object in the object feature data in the image data corresponding to each first image and / or the background brightness value in the background feature data. For example: based on the brightness value corresponding to the first object in the object feature data corresponding to the i-th first image and the brightness value corresponding to the first object in the object feature data corresponding to the i-1-th first image, the first brightness change value of the i-th first image relative to the i-1-th first image is determined, that is, the first brightness change value corresponding to the acquisition timestamp t_i corresponding to the i-th first image.

[0128] Based on the data corresponding to the second sensor in the sensor data corresponding to the multiple acquisition timestamps, the second brightness change value corresponding to each acquisition timestamp is determined. For example: based on the data corresponding to the second sensor corresponding to the acquisition timestamp t_i and the data corresponding to the second sensor corresponding to the acquisition timestamp t_i-1, the second brightness change value of the acquisition timestamp t_i relative to the acquisition timestamp t_i-1 is determined, that is, the second brightness change value corresponding to the acquisition timestamp t_i.

[0129] Afterwards, by determining whether the brightness value change indicated by the first brightness change value corresponding to the acquisition timestamp corresponding to each first image matches the brightness value change indicated by the second brightness change value corresponding to the acquisition timestamp t_i, and whether the movement trend of the mobile terminal indicated by the image posture data corresponding to the multiple acquisition timestamps and the movement trend of the mobile terminal indicated by the posture data in the sensor data corresponding to the multiple acquisition timestamps match, it is jointly determined whether the image acquisition process involves injection attack behavior to better improve the recognition accuracy of injection attack behavior.

[0130] Exemplarily, after determining that the brightness value change indicated by the first brightness change value corresponding to the acquisition timestamp corresponding to each first image matches the brightness value change indicated by the second brightness change value corresponding to the acquisition timestamp t_i, and determining that the movement trend of the mobile terminal indicated by the image posture data corresponding to multiple acquisition timestamps matches the movement trend of the mobile terminal indicated by the posture data in the sensor data corresponding to multiple acquisition timestamps, it is determined that the image acquisition process involves an injection attack behavior; if any aspect does not match, it can be determined that the image acquisition process involves an injection attack behavior.

[0131] In some other possible examples, a pre-trained machine learning network may be combined to determine whether the image acquisition process involves injection attack behavior based on the first data and the image data. Specifically, in step S230, step 21 may be specifically included:

[0132] In step 21, the first data and the image data are used to determine whether the image acquisition process involves injection attack behavior through an injection attack identification network, wherein the injection attack identification network is a network that is pre-trained based on sample data corresponding to each sample acquisition process and its corresponding label data, the sample data includes sample sensor data and sample image data corresponding to the corresponding sample acquisition process, and the label data is used to indicate whether the corresponding sample acquisition process involves injection attack behavior.

[0133] In some possible examples, the first data includes sensor data corresponding to a plurality of acquisition timestamps, and the image data includes image data obtained by identifying first images corresponding to a plurality of acquisition timestamps; the injection attack recognition model includes a matching degree recognition subnetwork;

[0134] like Fig. 6A As shown, in step 21, steps 211-214 may be included:

[0135] In step 211, based on the sensor data corresponding to each acquisition timestamp and the sensor data corresponding to the previous acquisition timestamp, the first change situation corresponding to each acquisition timestamp is determined. In this step, the acquisition timestamp t_i among the acquisition timestamps is taken as an example for description. Specifically, the difference between the sensor data corresponding to the acquisition timestamp t_i and the sensor data corresponding to the acquisition timestamp t_i-1 can be calculated as the first difference corresponding to the acquisition timestamp t_i, and based on the first difference corresponding to the acquisition timestamp t_i, the first change situation corresponding to the acquisition timestamp t_i is determined, wherein the first change situation corresponding to the acquisition timestamp t_i includes the first difference corresponding to the acquisition timestamp t_i.

[0136] In some other examples, the first change situation corresponding to each acquisition timestamp can also be determined based on the sensor data and acquisition mode data corresponding to each acquisition timestamp and the sensor data and acquisition mode data corresponding to the previous acquisition timestamp. The implementation principle of determining the first change situation corresponding to each acquisition timestamp is similar to the implementation principle of the aforementioned step 211, and the implementation process thereof can refer to the implementation process of the aforementioned step 211, which will not be described in detail here.

[0137] In step 212, based on the image data corresponding to each acquisition timestamp and the image data corresponding to the previous acquisition timestamp, the second change situation corresponding to each acquisition timestamp is determined. Exemplarily, the multi-dimensional feature data included in the image data is quantized data. In this step, the acquisition timestamp t_i in each acquisition timestamp is taken as an example for explanation. Specifically, the difference between the image data corresponding to the acquisition timestamp t_i and the image data corresponding to the acquisition timestamp t_i-1 can be calculated as the second difference corresponding to the acquisition timestamp t_i. Based on the second difference corresponding to the acquisition timestamp t_i, the second change situation corresponding to the acquisition timestamp t_i is determined, wherein the second change situation corresponding to the acquisition timestamp t_i includes the second difference corresponding to the acquisition timestamp t_i.

[0138] In step 213, based on the first change situation and the second change situation corresponding to each acquisition time stamp, a predicted matching value between the sensor data and the image data corresponding to each acquisition time stamp is obtained through a matching recognition subnetwork.

[0139] In some examples, the matching degree recognition subnetwork includes an encoder and a classifier. Accordingly, the first change situation and the second change situation corresponding to each acquisition timestamp can be spliced ​​to obtain the spliced ​​change situation corresponding to each acquisition timestamp. For example: the first change situation X1t_i corresponding to the acquisition timestamp t_i is expressed as [a, b, c, d], and the second change situation X2t_i corresponding to the acquisition timestamp t_i is expressed as [e, f, g], and the obtained spliced ​​change situation X1t_i+X2t_i corresponding to the acquisition timestamp t_i can be expressed as [a, b, c, d, e, f, g].

[0140] Afterwards, the splicing changes corresponding to each acquisition timestamp are input into the encoder of the matching recognition subnetwork, so as to process the splicing changes corresponding to each acquisition timestamp through the encoder to obtain the encoding representation corresponding to each acquisition timestamp (hereinafter referred to as the first encoding representation), and then the first encoding representation corresponding to each acquisition timestamp is input into the classifier of the matching recognition subnetwork, so as to process the first encoding representation corresponding to each acquisition timestamp through the classifier to obtain the predicted matching value between the sensor data and the image data corresponding to each acquisition timestamp.

[0141] In some other examples, the matching degree recognition subnetwork may include two encoders and a classifier, wherein one encoder corresponds to the first change situation, referred to as the first encoder; and one encoder corresponds to the second change situation, referred to as the second encoder. Accordingly, the first change situation corresponding to each acquisition timestamp is respectively input into the first encoder to obtain the encoding representation of the first change situation corresponding to each acquisition timestamp (hereinafter referred to as the second encoding representation); and the second change situation corresponding to each acquisition timestamp is respectively input into the second encoder to obtain the encoding representation of the second change situation corresponding to each acquisition timestamp (hereinafter referred to as the third encoding representation). Afterwards, the second encoding representation and the third encoding representation corresponding to each acquisition timestamp are spliced ​​to obtain the spliced ​​encoding representation corresponding to each acquisition timestamp. Next, the spliced ​​encoding representation corresponding to each acquisition timestamp is input into the classifier to obtain the predicted matching value between the sensor data and the image data corresponding to each acquisition timestamp.

[0142] Among them, the implementation principle of splicing the second coding representation and the third coding representation corresponding to each acquisition timestamp is the same as the implementation principle of splicing the first change situation and the second change situation corresponding to each acquisition timestamp. For its implementation process, refer to the aforementioned implementation process of splicing the first change situation and the second change situation corresponding to each acquisition timestamp, and will not be repeated here.

[0143] In some implementations, the matching degree recognition subnetwork can be implemented by any machine learning network, such as but not limited to a transformer-based network, a recognition network based on an SVM (Support Vector Machine) algorithm, a recognition network based on a random forest algorithm, and a recognition network based on a boost algorithm.

[0144] In step 214, based on each predicted matching value, it is determined whether the image acquisition process involves injection attack behavior. In some examples, if the number of predicted matching values ​​below a preset threshold in each predicted matching value exceeds a first preset number, or the ratio between the number of predicted matching values ​​below a preset threshold in each predicted matching value and the total number of predicted matching values ​​exceeds a preset ratio, it can be determined that the image acquisition process involves injection attack behavior. Otherwise, it is determined that the image acquisition process does not involve injection attack behavior.

[0145] In some other examples, if a second preset number of predicted matching values ​​lower than a preset threshold value appear continuously in each predicted matching value, it can be determined that the image acquisition process involves injection attack behavior. If a second preset number of predicted matching values ​​lower than a preset threshold value does not appear continuously in each predicted matching value, and the number of predicted matching values ​​lower than the preset threshold value in each predicted matching value does not exceed a third preset number, it can be determined that the image acquisition process does not involve injection attack behavior.

[0146] Considering that the first object is not included in the first image corresponding to the image acquisition process, or the first object is included and the first object is a human face, or the first object is included and the first object is a card in a handheld state, the first object is prone to be moved, and the first object is moved, and the mobile terminal is also moved, which will, to a certain extent, lead to the movement of the first object presented by the image, and the movement of the mobile terminal determined and indicated is not accurate enough. In view of this, in order to ensure the accuracy of the recognition result of the injection attack behavior, in some possible examples, the matching degree recognition subnetwork includes a first matching degree recognition subnetwork; the first matching degree recognition subnetwork is used to identify the matching degree value between the background feature data in the image data and the sensor data (and acquisition method data) in the first data.

[0147] Correspondingly, such as Figure 7 As shown, if the first image indicates that it does not include the first object or the first object is in a handheld state or the first object is a human face, in step 212, it may specifically include: based on the background feature data corresponding to each acquisition timestamp and the background feature data corresponding to its previous acquisition timestamp, determining the second change situation corresponding to each acquisition timestamp. Exemplarily, in this step, the acquisition timestamp t_i among the acquisition timestamps is taken as an example for explanation. Specifically, the difference between the background feature data corresponding to the acquisition timestamp t_i and the background feature data corresponding to the acquisition timestamp t_i-1 may be calculated as the third difference corresponding to the acquisition timestamp t_i, and based on the third difference corresponding to the acquisition timestamp t_i, the second change situation corresponding to the acquisition timestamp t_i is determined, wherein the second change situation corresponding to the acquisition timestamp t_i includes the third difference corresponding to the acquisition timestamp t_i.

[0148] Accordingly, in step 213, it may specifically include: based on the first change situation and the second change situation corresponding to each acquisition time stamp, obtaining the predicted matching value corresponding to each acquisition time stamp through the first matching recognition sub-network. In this step, the first change situation and the second change situation corresponding to each acquisition time stamp may be input into the first matching recognition sub-network, so as to process the first change situation and the second change situation corresponding to each acquisition time stamp through the first matching recognition sub-network, so as to obtain the predicted matching value corresponding to each acquisition time stamp.

[0149] In some examples, the first match recognition subnetwork includes an encoder and a classifier; or the first match recognition subnetwork includes two encoders (one corresponding to the first change situation and the other corresponding to the second change situation object) and a classifier. The implementation principle of the first match recognition subnetwork processing data is similar to the implementation principle of the aforementioned match recognition subnetwork processing data. The implementation process of the first match recognition subnetwork processing data can refer to the implementation process of the aforementioned match recognition subnetwork processing data, which will not be repeated here.

[0150] In some other possible examples, the aforementioned matching degree recognition subnetwork includes a second matching degree recognition subnetwork; the second matching degree recognition subnetwork is used to recognize the matching degree value between the object feature data in the image data and the sensor data (and acquisition method data) in the first data; accordingly, Figure 7 As shown, if the first image indicates that the first object is in a non-handheld state (for example, placed on a plane, such as a card on a desktop, then the first object can be considered to be in a static state, and the movement of the first object presented in the image can be determined to be relatively accurate), in step 212, it can specifically include: based on the object feature data corresponding to each acquisition timestamp and the object feature data corresponding to its previous acquisition timestamp, determining the second change corresponding to each acquisition timestamp, wherein the object feature data is identified based on the first object in the first image corresponding to each acquisition timestamp.

[0151] Exemplarily, in this step, the acquisition timestamp t_i among each acquisition timestamp is taken as an example for explanation. Specifically, the difference between the object feature data corresponding to the acquisition timestamp t_i and the object feature data corresponding to the acquisition timestamp t_i-1 can be calculated as the fourth difference corresponding to the acquisition timestamp t_i, and based on the fourth difference corresponding to the acquisition timestamp t_i, the second change situation corresponding to the acquisition timestamp t_i is determined, wherein the second change situation corresponding to the acquisition timestamp t_i includes the fourth difference corresponding to the acquisition timestamp t_i.

[0152] Then, in step 213, it may specifically include: based on the first change situation and the second change situation corresponding to each acquisition time stamp, obtaining the predicted matching value corresponding to each acquisition time stamp through the second matching recognition sub-network. In this step, the first change situation and the second change situation corresponding to each acquisition time stamp may be input into the second matching recognition sub-network, so as to process the first change situation and the second change situation corresponding to each acquisition time stamp through the second matching recognition sub-network, so as to obtain the predicted matching value corresponding to each acquisition time stamp.

[0153] In some examples, the second match recognition subnetwork includes an encoder and a classifier; or the second match recognition subnetwork includes two encoders (one corresponding to the first change situation and the other corresponding to the second change situation object) and a classifier. The implementation principle of the second match recognition subnetwork processing data is similar to the implementation principle of the aforementioned match recognition subnetwork processing data. The implementation process of the second match recognition subnetwork processing data can refer to the implementation process of the aforementioned match recognition subnetwork processing data, which will not be repeated here.

[0154] In some other possible examples, such as Figure 6B and Figure 7 As shown, the injection attack identification network includes a classification subnetwork; accordingly, in step 21, steps 31-32 may be included:

[0155] In step 31, the first data and the image data are input into the classification subnetwork to obtain a predicted classification result.

[0156] Exemplarily, the classification subnetwork can be any network that can process time series data. For example, the classification subnetwork can be a recognition network based on a transformer structure, a recognition network based on a long short-term memory network, or a recognition network based on a convolutional neural network, or a recognition network based on an RNN (Recurrent Neural Network).

[0157] Some possible examples include Figure 7 As shown, the object feature data in the first data and the image data may be input into the classification sub-network to obtain a prediction classification result that can indicate whether the image acquisition process involves injection attack behavior. The background feature data in the first data and the image data may be input into the classification sub-network to obtain a prediction classification result that can indicate whether the image acquisition process involves injection attack behavior. The first data and the object feature data and the background feature data in the image data may be input into the classification sub-network to obtain a prediction classification result that can indicate whether the image acquisition process involves injection attack behavior.

[0158] The first data may include sensor data and / or acquisition method data.

[0159] In some possible examples, the aforementioned classification subnetwork may include an extraction layer and a classification layer; in step 31, steps 311-312 may be included:

[0160] In step 311, an intermediate representation is obtained based on the first data and the image data through an extraction layer. In this step, the extraction layer can be any extraction layer that can process time series data. Exemplarily, the extraction layer can be an extraction layer based on a transformer structure, an extraction layer based on a long short-term memory network, an extraction layer based on a convolutional neural network, or an extraction layer based on an RNN.

[0161] In some possible implementations, the mobile terminal may splice the first data and the image data to obtain the aforementioned spliced ​​data, and then input the spliced ​​data into the extraction layer so as to perform feature extraction on the spliced ​​data through the extraction layer to obtain an intermediate representation.

[0162] Exemplarily, the first data may include sensor data (and acquisition method data) corresponding to multiple acquisition timestamps, and the image data may include each image data identified from each first image corresponding to the multiple acquisition timestamps. The stitched data includes stitched data corresponding to each acquisition timestamp. Taking the acquisition timestamp t_i in each acquisition timestamp as an example, the stitching process is explained. Specifically, the sensor data (and acquisition method data) Ci corresponding to the acquisition timestamp t_i in the first data is expressed as [q, w, e, r], and the image data Ti corresponding to the acquisition timestamp t_i in the image data is expressed as [t, y, u, i]. Correspondingly, the stitched data Ci+Ti corresponding to the acquisition timestamp t_i can be expressed as [q, w, e, r, t, y, u, i].

[0163] Then, in step 312, the aforementioned intermediate representation is input into the classification layer to obtain a predicted classification result indicating whether the image acquisition process involves injection attack behavior. In this step, the mobile terminal (target program) can input the intermediate representation into the classification layer to process the intermediate representation through the classification layer to obtain a predicted classification result indicating whether the image acquisition process involves injection attack behavior.

[0164] Exemplarily, the classification layer can be implemented by a multi-layer perceptron or other classifiers.

[0165] In some examples, the classification layer may output a probability value indicating that the image acquisition process involves an injection attack behavior, and the probability value may represent a predicted classification result of whether the image acquisition process involves an injection attack behavior. Exemplarily, if the probability value is greater than or equal to a preset probability threshold, it may be considered that the predicted classification result indicates that the image acquisition process involves an injection attack behavior; if the probability value is less than the preset probability threshold, it may be considered that the predicted classification result indicates that the image acquisition process involves an injection attack behavior.

[0166] After the prediction classification result is obtained, in step 32, it is determined whether the image acquisition process involves injection attack behavior based on the prediction classification result. In some possible examples, if the prediction classification result indicates that the image acquisition process involves injection attack behavior, it is determined that the image acquisition process involves injection attack behavior, and if the prediction classification result indicates that the image acquisition process does not involve injection attack behavior, it is determined that the image acquisition process does not involve injection attack behavior.

[0167] Some possible examples include Figure 7 As shown, the injection attack identification network includes the aforementioned matching identification subnetwork and classification subnetwork (and the subsequent time coding subnetwork corresponding to the classification subnetwork), so as to jointly determine the final result of whether the image acquisition process involves the injection attack behavior by combining the predicted matching values ​​obtained by the matching identification subnetwork, the result of whether the image acquisition process involves the injection attack behavior, and the predicted classification result obtained by the classification subnetwork. For example, in order to better ensure the privacy information and property security of the user, it can be that among the predicted matching values ​​obtained by the matching identification subnetwork, the result of whether the image acquisition process involves the injection attack behavior, and the predicted classification result obtained by the classification subnetwork, there is at least one result indicating that the image acquisition process involves the injection attack behavior, and the final result is determined to be that the image acquisition process involves the injection attack behavior. For another example, it can also be set that among the predicted matching values ​​obtained by the matching identification subnetwork, the result of whether the image acquisition process involves the injection attack behavior, and the predicted classification result obtained by the classification subnetwork, there is at least one result indicating that the image acquisition process does not involve the injection attack behavior, and the final result is determined to be that the image acquisition process does not involve the injection attack behavior.

[0168] In some possible examples, the aforementioned injection attack identification network includes a time coding subnetwork and a classification subnetwork: Accordingly, in step 21, steps 41-43 may be included:

[0169] In step 41, each acquisition time stamp is input into the time coding sub-network to obtain the time coding data corresponding to each acquisition time stamp.

[0170] Among them, the time coding subnetwork can encode each acquisition timestamp to convert each acquisition timestamp into a vector representation that can capture the complexity of the time dimension. Specifically, the mobile terminal can input each acquisition timestamp into the time coding subnetwork, and process each acquisition timestamp through the time coding subnetwork to obtain the time coding data corresponding to each acquisition timestamp. Exemplarily, the time coding subnetwork can be implemented as a Time2vec algorithm. The Time2Vec algorithm is a special encoding method that can capture the periodicity and pattern of time series data. It can encode the timestamp to capture the complexity of the time dimension therein and obtain time coding data (i.e., vector representation) that can characterize the complexity of the time dimension. Specifically, the time2vec algorithm can be expressed by the following formula:

[0171]

[0172] Wherein, t2v(τ) represents the time coding data corresponding to each acquisition timestamp τ; t2v(τ)[k] represents the kth element in the time coding data corresponding to the acquisition timestamp τ, ω k and is a trainable network parameter; F(·) is a set periodic activation function, which can be set as a sine function, and exemplarily, the F(·) can also be set as a cosine function; K is a preset hyperparameter.

[0173] After the mobile terminal obtains the first data and the image data, it inputs each acquisition time stamp into the time coding sub-network, so that the time coding sub-network obtains the time coding data corresponding to each acquisition time stamp through the above formula (1). It can be understood that the time coding data corresponding to a single acquisition time stamp is a K+1-dimensional vector.

[0174] In some other examples, the time coding subnetwork may also be implemented by other algorithms that can encode timestamp data, such as a coding algorithm based on a sine function and a coding algorithm based on a cosine function.

[0175] In step 42, the first data, the image data and the time coding data corresponding to each acquisition time stamp are input into the classification sub-network to obtain a predicted classification result.

[0176] Exemplarily, the classification subnetwork includes an extraction layer and a classification layer, wherein the extraction layer may be an extraction layer based on a transformer structure, an extraction layer based on a long short-term memory network, an extraction layer based on a convolutional neural network, or an extraction layer based on an RNN. In this step, the mobile terminal (target program) combines the sensor data (and acquisition mode data) corresponding to each acquisition timestamp in the first data, the image data corresponding to each acquisition timestamp in the image data, and the time coding data corresponding to each acquisition timestamp to obtain an intermediate data sequence. For example, for any acquisition timestamp j in each acquisition timestamp in the first data, assuming that the corresponding sensor data (and acquisition mode data) is represented as x1j (wherein, x1j may be an array), assuming that the corresponding image data is represented as x2j, and the time coding data is represented as x3j, the sensor data (and acquisition mode data), image data and time coding data corresponding to the acquisition timestamp j are combined, and the intermediate data corresponding to the acquisition timestamp j obtained can be represented as [x1j, x2j, x3j]. In this way, the intermediate data corresponding to each acquisition timestamp is obtained to form an intermediate data sequence.

[0177] Afterwards, the intermediate data sequence is input into the extraction layer of the classification subnetwork to process the intermediate data sequence through the extraction layer to obtain an intermediate representation.

[0178] In some possible implementations, the extraction layer of the classification subnetwork can be an extraction layer based on a transformer structure; the extraction layer is used to perform convolution processing and processing based on a self-attention mechanism on the input; accordingly, the aforementioned process of obtaining the intermediate representation can include: inputting the intermediate data sequence into the extraction layer, so that the extraction layer performs convolution processing on the intermediate data sequence to obtain convolution processing result data, and processing the convolution processing result data based on the self-attention mechanism to obtain the intermediate representation.

[0179] In the above steps, the extraction layer performs convolution processing on the intermediate data sequence, which can realize the feature extraction of local context information in the intermediate data sequence. Afterwards, based on the self-attention mechanism, the convolution processing result data is processed to realize the extraction of the overall relationship of the data, and obtain an intermediate representation that is more helpful to improve the classification results, so as to better enhance the modeling ability of the injection classification subnetwork for local context information in time series data, thereby reducing the influence of abnormal data in the time series data (i.e., the intermediate data sequence) on subsequent classification results, and improving the accuracy of the classification results, that is, achieving the recognition accuracy of whether the image acquisition process involves injection attack behavior.

[0180] Next, the aforementioned intermediate representation is input into the classification layer to obtain a predicted classification result indicating whether the image acquisition process involves injection attack behavior. In this step, the aforementioned intermediate representation is input into the classification layer so that the classification layer processes the intermediate representation to obtain a predicted classification result indicating whether the image acquisition process involves injection attack behavior.

[0181] Then, in step 43, based on the predicted classification result, it is determined whether the image acquisition process involves injection attack behavior. The implementation principle of this step is similar to the implementation principle of the aforementioned step 32. The implementation process can refer to the implementation process of step 32, and will not be repeated here.

[0182] In the above process, the time information (i.e., each acquisition timestamp) is encoded into the intermediate representation through the time encoding subnetwork and the classification subnetwork, so that the classification subnetwork can better understand and process the time series data, i.e., the first data and the image data, so as to obtain a more accurate classification result based on the intermediate representation that combines the time information and the feature information corresponding to each time information (i.e., the sensor data, acquisition method data and image data corresponding to each acquisition timestamp).

[0183] The above embodiment provides a new method for identifying injection attack behavior. Considering that in a real image acquisition process (i.e., an image acquisition process that does not involve injection attack behavior), the movement of the mobile terminal in the image acquisition process (and changes in ambient light) is associated with the movement of the first object in the image sequence acquired in the image acquisition process (and changes in image brightness). For example, the mobile terminal moves to the right relative to the first object, and accordingly, the image sequence will indicate that the first object (and / or the background in the image) moves to the left relative to the mobile terminal. For another example, the mobile terminal approaches the first object, and accordingly, the image sequence will indicate that the first object (and / or the background in the image) becomes larger. In other words, the image sequence can also indicate the movement of the mobile terminal. For another example, when the ambient light in the environment where the mobile terminal is located becomes weaker, the image brightness value in the corresponding image sequence will become smaller. In view of this, in the implementation mode provided in the present specification, by identifying and determining the correlation between the first data corresponding to the image acquisition process, namely the sensor data (which can characterize the movement of the mobile terminal and / or the change of ambient light during the image acquisition process), and the image data corresponding to the image acquisition process (which can also indicate the movement of the mobile terminal and / or the change of ambient light), for example, determining whether the movement of the mobile terminal indicated by the two is consistent (and determining whether the changes of ambient light indicated by the two are consistent), it is determined whether the image acquisition process involves an injection attack behavior, thereby realizing the identification of the injection attack behavior.

[0184] Furthermore, considering the obvious difference in the movement of the mobile terminal between the image acquisition process involving injection attack behavior and the real image acquisition process (i.e., the image acquisition process not involving injection attack behavior), specifically, in the real image acquisition process, the mobile terminal generally moves following the user acquisition behavior in the acquisition process (i.e., the user controls the acquisition behavior of the mobile terminal according to the image acquisition instruction), and accordingly, in the real image acquisition process, the movement of the mobile terminal is consistently associated with the user acquisition behavior; while in the image acquisition process involving injection attack behavior, the mobile terminal is generally placed flat and / or motionless, and accordingly, the movement of the mobile terminal is not consistently associated with the user acquisition behavior. In view of the above situation, the first data corresponding to the image acquisition process also includes acquisition method data for indicating that the user adopts a specific acquisition method, and the correlation relationship between the sensor data and the acquisition method data, as well as the correlation relationship between the sensor data, the acquisition method data and the image data, is combined to identify whether the image acquisition process involves injection attack behavior, thereby better realizing the identification of injection attack behavior.

[0185] The foregoing describes certain embodiments of the present specification, and other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims may be performed in an order different from that in the embodiments, and the desired results may still be achieved. In addition, the processes depicted in the accompanying drawings do not necessarily have to be performed in the specific order or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0186] Corresponding to the above method embodiment, the present specification embodiment provides an injection attack behavior identification device 800, whose schematic block diagram is as follows: Figure 8 As shown, it includes: a first acquisition module 810, configured to acquire first data corresponding to an image acquisition process of a mobile terminal for a first object, wherein the first data includes sensor data, the sensor data corresponds to a sensor in the mobile terminal, and the sensor includes a first sensor for sensing the movement of the mobile terminal; a second acquisition module 820, configured to acquire image data corresponding to the image acquisition process, wherein the image data is obtained from a first image recognition corresponding to the image acquisition process, and the image data is used to indicate the movement of the mobile terminal; a first determination module 830, configured to use the first data and the image data to determine whether the image acquisition process involves an injection attack behavior.

[0187] In some possible examples, the first object is a card or a human face.

[0188] In some possible examples, the first data also includes acquisition method data, and the acquisition method data is used to instruct the user to adopt a specific acquisition method during the image acquisition process.

[0189] In some possible examples, the acquisition method data includes: data indicating a frontal acquisition method and / or data indicating a non-frontal acquisition method, and the non-frontal acquisition method includes: a random multi-tilt angle acquisition method.

[0190] In some possible examples, it also includes: a generation module (not shown in the figure), configured to randomly generate a first instruction during the image acquisition process, the first instruction being used to instruct a preset vibrator in the mobile terminal to vibrate; the first data also includes: data indicating whether the mobile terminal is vibrating.

[0191] In some possible examples, the mobile terminal is provided with multiple types of sensors; when the first data is real data corresponding to the image acquisition process, the sensor data includes data directly measured by the multiple types of sensors and / or posture data of the mobile terminal determined based on the data measured by the first sensor.

[0192] In some possible examples, the first data includes sensor data corresponding to multiple acquisition timestamps; and the image data includes each image data identified from the first images corresponding to the multiple acquisition timestamps.

[0193] In some possible examples, the second acquisition module 820 is specifically configured to detect object feature data corresponding to the first object from the first image when it is detected that the first image contains the first object, and include the object feature data into the image data corresponding to the first image.

[0194] In some possible examples, the object feature data includes at least one of the following types of data: regional position information of the image area where the first object is located in the first image, the Euler angle, clarity value and brightness value corresponding to the first object, and a first estimated distance between the first object and the mobile terminal, wherein the first estimated distance is determined based on the proportion of the image area where the first object is located in the first image, and the Euler angle, clarity value and brightness value corresponding to the first object are determined based on the first image and a pre-trained attribute recognition network.

[0195] In some possible examples, the second acquisition module 820 is further configured to:

[0196] When it is detected that the first image does not contain the first object, background feature data is extracted from the first image and incorporated into image data corresponding to the first image, wherein the background feature data is used to calculate the motion of the first image relative to its previous frame image.

[0197] In some possible examples, the second acquisition module 820 is further configured to:

[0198] In the case where it is detected that the first image contains the first object, subtracting the image area where the first object is located from the first image to obtain a background area image corresponding to the first image;

[0199] Background feature data is extracted from the background area image and incorporated into the image data corresponding to the first image, wherein the background feature data is used to calculate the motion of the first image relative to its previous frame image.

[0200] In some possible examples, the first determination module 830 is specifically configured to use the first data and the image data to determine whether the image acquisition process involves injection attack behavior through an injection attack identification network, wherein the injection attack identification network is a network that is pre-trained based on sample data corresponding to each sample acquisition process and its corresponding label data, the sample data includes sample sensor data and sample image data corresponding to the corresponding sample acquisition process, and the label data is used to indicate whether the corresponding sample acquisition process involves injection attack behavior.

[0201] In some possible examples, the first data includes sensor data corresponding to a plurality of acquisition timestamps, the image data includes each image data identified from each first image corresponding to a plurality of acquisition timestamps; the injection attack recognition model includes a matching degree recognition subnetwork;

[0202] The first determination module 830 includes: a first determination unit (not shown in the figure), configured to determine a first change situation corresponding to each acquisition timestamp based on the sensor data corresponding to each acquisition timestamp and the sensor data corresponding to the previous acquisition timestamp;

[0203] A second determining unit (not shown in the figure) is configured to determine a second change situation corresponding to each acquisition timestamp based on the image data corresponding to each acquisition timestamp and the image data corresponding to the previous acquisition timestamp;

[0204] A first obtaining unit (not shown in the figure) is configured to obtain a predicted matching value between the sensor data and the image data corresponding to each acquisition time stamp through the matching recognition subnetwork based on the first change situation and the second change situation corresponding to each acquisition time stamp;

[0205] The third determination unit (not shown in the figure) is configured to determine whether the image acquisition process involves injection attack behavior based on each predicted matching degree value.

[0206] In some possible examples, the match degree identification subnetwork includes a first match degree identification subnetwork;

[0207] If the first image indicates that it does not include the first object or the first object is in a handheld state, the second determining unit is specifically configured to determine a second change situation corresponding to each acquisition timestamp based on the background feature data corresponding to each acquisition timestamp and the background feature data corresponding to a previous acquisition timestamp;

[0208] The first obtaining unit is specifically configured to obtain the predicted matching degree value corresponding to each acquisition timestamp through the first matching degree identification subnetwork based on the first change situation and the second change situation corresponding to each acquisition timestamp.

[0209] In some possible examples, the match degree identification subnetwork includes a second match degree identification subnetwork;

[0210] If the first image indicates that the first object is in a non-handheld state, the second determining unit is specifically configured to determine a second change situation corresponding to each acquisition timestamp based on object feature data corresponding to each acquisition timestamp and object feature data corresponding to a previous acquisition timestamp, wherein the object feature data is obtained by identifying the first object in the first image corresponding to each acquisition timestamp;

[0211] The first obtaining unit is specifically configured to obtain the predicted matching degree value corresponding to each acquisition timestamp through the second matching degree identification subnetwork based on the first change situation and the second change situation corresponding to each acquisition timestamp.

[0212] In some possible examples, the injection attack identification network includes a classification subnetwork;

[0213] The first determination module 830 includes: a second obtaining unit (not shown in the figure), configured to input the first data and the image data into the classification subnetwork to obtain a predicted classification result;

[0214] The fourth determination unit (not shown in the figure) is configured to determine whether the image acquisition process involves injection attack behavior based on the predicted classification result.

[0215] In some possible examples, the injection attack identification network further includes a time coding subnetwork:

[0216] The first determination module 830 further includes: a third obtaining unit (not shown in the figure), configured to input each acquisition timestamp into the time coding subnetwork to obtain time coding data corresponding to each acquisition timestamp before inputting the first data and the image data into the classification subnetwork to obtain the predicted classification result;

[0217] The second obtaining unit is configured to input the first data, the image data and the time encoding data corresponding to each acquisition timestamp into the classification subnetwork to obtain the predicted classification result.

[0218] The above device embodiments correspond to the method embodiments. For specific descriptions, please refer to the description of the method embodiments, which will not be repeated here. The device embodiments are obtained based on the corresponding method embodiments and have the same technical effects as the corresponding method embodiments. For specific descriptions, please refer to the corresponding method embodiments.

[0219] The embodiments of the present specification also provide a computer-readable storage medium on which a computer program is stored. When the computer program is executed in a computer, the computer is caused to execute the injection attack behavior identification method provided in the present specification.

[0220] The embodiment of the present specification also provides a computing device, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method for identifying injection attack behavior provided in the present specification is implemented.

[0221] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the storage medium and computing device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0222] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the embodiments of the present invention may be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions may be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.

[0223] The specific implementation methods described above further describe the purpose, technical solutions and beneficial effects of the embodiments of the present invention in detail. It should be understood that the above description is only a specific implementation method of the embodiments of the present invention and is not intended to limit the scope of protection of the present invention. Any modification, equivalent replacement, improvement, etc. made on the basis of the technical solution of the present invention shall be included in the scope of protection of the present invention.

Claims

1. A method for identifying injection attack behavior, comprising: Acquire first data corresponding to an image acquisition process of a mobile terminal for a first object, wherein the first data includes sensor data, the sensor data corresponds to a sensor in the mobile terminal, and the sensor includes a first sensor for sensing a motion of the mobile terminal; Acquire image data corresponding to the image acquisition process, wherein the image data is obtained from the first image recognition corresponding to the image acquisition process, and the image data is used to indicate the movement of the mobile terminal; Using the first data and the image data, it is determined whether the image acquisition process involves injection attack behavior.

2. The method of claim 1, wherein: The first object is a card or a face.

3. The method of claim 1, wherein: The first data also includes acquisition method data, and the acquisition method data is used to instruct the user to adopt a specific acquisition method during the image acquisition process.

4. The method of claim 3, wherein: The acquisition mode data includes: data indicating a frontal acquisition mode and / or data indicating a non-frontal acquisition mode, and the non-frontal acquisition mode includes: a random multi-tilt angle acquisition mode.

5. The method of claim 1, further comprising: During the image acquisition process, randomly generating a first instruction, the first instruction being used to instruct a preset vibrator in the mobile terminal to vibrate; The first data also includes: data indicating whether the mobile terminal vibrates.

6. The method of claim 1, wherein: The mobile terminal is provided with multiple types of sensors; In the case where the first data is real data corresponding to the image acquisition process, the sensor data includes data directly measured by the multiple types of sensors and / or posture data of the mobile terminal determined based on the data measured by the first sensor.

7. The method of claim 1, wherein: The first data includes sensor data corresponding to multiple acquisition timestamps in the image acquisition process; the image data includes each image data identified from the first images corresponding to the multiple acquisition timestamps.

8. The method of claim 1, wherein: The obtaining of image data corresponding to the image acquisition process includes: In the case where it is detected that the first image includes the first object, object feature data corresponding to the first object is detected from the first image and included in the image data corresponding to the first image.

9. The method of claim 8, wherein: The object feature data includes at least one of the following types of data: regional position information of the image area where the first object is located in the first image, the Euler angle, clarity value and brightness value corresponding to the first object, and a first estimated distance between the first object and the mobile terminal, wherein the first estimated distance is determined based on the proportion of the image area where the first object is located in the first image, and the Euler angle, clarity value and brightness value corresponding to the first object are determined based on the first image and a pre-trained attribute recognition network.

10. The method of claim 1, further comprising: When it is detected that the first image does not contain the first object, background feature data is extracted from the first image and incorporated into image data corresponding to the first image, wherein the background feature data is used to calculate the motion of the first image relative to its previous frame image.

11. The method of claim 8, wherein: The method further comprises: In the case where it is detected that the first image contains the first object, subtracting the image area where the first object is located from the first image to obtain a background area image corresponding to the first image; Background feature data is extracted from the background area image and incorporated into the image data corresponding to the first image, wherein the background feature data is used to calculate the motion of the first image relative to its previous frame image.

12. The method according to any one of claims 1 to 11, wherein: The determining whether the image acquisition process involves injection attack behavior includes: Utilizing the first data and the image data, an injection attack identification network is used to determine whether the image acquisition process involves injection attack behavior, wherein the injection attack identification network is a network that is pre-trained based on sample data corresponding to each sample acquisition process and its corresponding label data, the sample data includes sample sensor data and sample image data corresponding to the corresponding sample acquisition process, and the label data is used to indicate whether the corresponding sample acquisition process involves injection attack behavior.

13. The method of claim 12, wherein: The first data includes sensor data corresponding to a plurality of acquisition timestamps, and the image data includes each image data identified from each first image corresponding to the plurality of acquisition timestamps; The injection attack identification model includes a matching degree identification sub-network; The determining whether the image acquisition process involves injection attack behavior includes: Determine a first change situation corresponding to each acquisition timestamp based on the sensor data corresponding to each acquisition timestamp and the sensor data corresponding to the previous acquisition timestamp; Determine a second change situation corresponding to each acquisition time stamp based on the image data corresponding to each acquisition time stamp and the image data corresponding to the previous acquisition time stamp; Based on the first change situation and the second change situation corresponding to each acquisition time stamp, obtaining the predicted matching value between the sensor data and the image data corresponding to each acquisition time stamp through the matching recognition subnetwork; Based on each predicted matching degree value, it is determined whether the image acquisition process involves injection attack behavior.

14. The method of claim 13, wherein: The matching degree identification sub-network includes a first matching degree identification sub-network; If the first image indicates that it does not include the first object or the first object is in a handheld state, determining the second change situation corresponding to each acquisition timestamp based on the image data corresponding to each acquisition timestamp and the image data corresponding to the previous acquisition timestamp thereof includes: Determine a second change situation corresponding to each acquisition time stamp based on the background feature data corresponding to each acquisition time stamp and the background feature data corresponding to the previous acquisition time stamp; The step of obtaining the predicted matching value between the sensor data and the image data corresponding to each acquisition timestamp includes: Based on the first change situation and the second change situation corresponding to each acquisition time stamp, the predicted matching degree value corresponding to each acquisition time stamp is obtained through the first matching degree identification sub-network.

15. The method of claim 13, wherein: The matching degree identification sub-network includes a second matching degree identification sub-network; If the first image indicates that the first object is in a non-handheld state, determining the second change situation corresponding to each acquisition timestamp based on the image data corresponding to each acquisition timestamp and the image data corresponding to the previous acquisition timestamp thereof includes: Determine a second change situation corresponding to each acquisition time stamp based on object feature data corresponding to each acquisition time stamp and object feature data corresponding to a previous acquisition time stamp, wherein the object feature data is obtained by identifying a first object in a first image corresponding to each acquisition time stamp; The step of obtaining the predicted matching value between the sensor data and the image data corresponding to each acquisition timestamp includes: Based on the first change situation and the second change situation corresponding to each acquisition time stamp, the predicted matching degree value corresponding to each acquisition time stamp is obtained through the second matching degree identification subnetwork.

16. The method of claim 12, wherein: The injection attack identification network includes a classification subnetwork; The determining whether the image acquisition process involves injection attack behavior includes: Inputting the first data and the image data into the classification subnetwork to obtain a predicted classification result; Based on the predicted classification result, it is determined whether the image acquisition process involves injection attack behavior.

17. The method according to claim 16, wherein the injection attack identification network further comprises a time coding subnetwork: Before inputting the first data and the image data into the classification subnetwork to obtain the predicted classification result, the method further includes: Input each acquisition time stamp into the time coding subnetwork to obtain time coding data corresponding to each acquisition time stamp; The obtained prediction classification result includes: The first data, the image data and the time coding data corresponding to each acquisition timestamp are input into the classification subnetwork to obtain the predicted classification result.

18. A device for identifying injection attack behavior, comprising: A first acquisition module is configured to acquire first data corresponding to an image acquisition process of a mobile terminal for a first object, wherein the first data includes sensor data, the sensor data corresponds to a sensor in the mobile terminal, and the sensor includes a first sensor for sensing a motion of the mobile terminal; A second acquisition module is configured to acquire image data corresponding to the image acquisition process, wherein the image data is obtained from the first image recognition corresponding to the image acquisition process, and the image data is used to indicate the movement of the mobile terminal; The first determination module is configured to use the first data and the image data to determine whether the image acquisition process involves injection attack behavior.

19. A computing device comprising a memory and a processor, wherein: The memory stores executable codes, and when the processor executes the executable codes, the method according to any one of claims 1 to 17 is implemented.

Images