Method and device for identifying injection attack behavior

Abstract

The embodiment of the invention provides an injection attack behavior identification method and device, and the method comprises the steps: in the image collection process of a mobile terminal, obtaining first sensor data and a first image frame corresponding to a first moment in the image collection process, the first sensor data corresponding to a sensor in the mobile terminal, and the first image frame corresponding to a second moment in the image collection process; the sensor is used for sensing movement of the mobile terminal; pre-stored second sensor data and second image frames corresponding to N second moments are obtained, and the N second moments are N moments before the first moment in the image acquisition process; and determining whether the image acquisition process relates to an injection attack behavior or not by using the first sensor data and the first image frame as well as the second sensor data and the second image frame corresponding to the N second moments, thereby realizing identification of the injection attack behavior.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method and device for identifying injection attack behavior. Background Art

[0002] With the development of artificial intelligence (AI) technology, content generation technology based on AI technology is becoming more and more mature, and the content it generates is becoming more and more realistic, and it is becoming increasingly difficult to distinguish its authenticity. For example, Deepfake technology is a technology that uses AI technology to generate seemingly real fake videos and / or fake audio; another example is AIGC (Artificial Intelligence Generated Content) technology, which is based on AI technology and generates new data (i.e., new content, such as text, images, audio and video) with generalization capabilities through learning and identifying existing data.

[0003] As the content generated by AI-based content generation technology is becoming more and more realistic and more and more difficult to distinguish its authenticity, in some scenarios, it is inevitable that criminals will use such technology to generate fake content for profit. For example, in the scenario of using the eKYC (electronic know your customer) system to authenticate users, it is easy for the black industry to use AI-based content generation technology to generate fake images or videos (including faces and / or certificates, etc.) and perform injection attacks in order to deceive the eKYC system to pass the verification; for example, in the scenario of using real-time communication software for real-time communication, it is easy for the black industry to use AI-based content generation technology to generate fake images or videos (including faces or other objects, etc.) and perform injection attacks in order to deceive the other party in real-time communication. Among them, in the above-mentioned injection attack process, the black industry generally takes advantage of the loopholes of the mobile terminal to inject the generated fake images or videos into the acquisition unit of the corresponding software or system instead of the real images or videos captured by the camera of the mobile terminal.

[0004] At present, due to the continuous development of AI-based content generation technology, the forged images (or videos) are becoming more and more realistic, making it increasingly difficult to identify and defend only from the images or videos themselves. In order to ensure the privacy information and / or property security of users in the above scenarios, it is crucial to provide a new method for identifying injection attack behaviors for defense. Summary of the invention

[0005] One or more embodiments of the present specification provide a method and device for identifying injection attack behaviors, so as to realize the identification of injection attack behaviors.

[0006] According to a first aspect, a method for identifying injection attack behavior is provided, comprising:

[0007] During an image acquisition process of a mobile terminal, first sensor data and a first image frame corresponding to a first moment in the image acquisition process are acquired, wherein the first sensor data corresponds to a sensor in the mobile terminal, and the sensor is used to sense a movement of the mobile terminal;

[0008] Acquire second sensor data and second image frames corresponding to respective N pre-stored second moments, wherein the N second moments are N moments before the first moment in the image acquisition process;

[0009] It is determined whether the image acquisition process involves injection attack behavior by using the first sensor data and the first image frame, and the second sensor data and the second image frame corresponding to each of the N second moments.

[0010] According to a second aspect, a device for identifying injection attack behavior is provided, comprising:

[0011] A first acquisition module is configured to acquire, during an image acquisition process of a mobile terminal, first sensor data and a first image frame corresponding to a first moment in the image acquisition process, wherein the first sensor data corresponds to a sensor in the mobile terminal, and the sensor is used to sense a movement of the mobile terminal;

[0012] A second acquisition module is configured to acquire second sensor data and second image frames corresponding to N pre-stored second moments, respectively, wherein the N second moments are N moments before the first moment in the image acquisition process;

[0013] The determination module is configured to determine whether the image acquisition process involves injection attack behavior by using the first sensor data and the first image frame, and the second sensor data and the second image frame corresponding to each of the N second moments.

[0014] According to a third aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed in a computer, the computer is caused to execute the method described in the first aspect.

[0015] According to a fourth aspect, a computing device is provided, comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method described in the first aspect is implemented.

[0016] According to the injection attack behavior identification method and device provided in the embodiments of the present specification, during the image acquisition process of the mobile terminal, the first sensor data and the first image frame corresponding to the first moment in the image acquisition process are obtained, and then the second sensor data and the second image frame corresponding to each of the N pre-stored second moments are obtained, wherein the N second moments are the first N moments of the first moment in the image acquisition process; then, the first sensor data and the first image frame, as well as the second sensor data and the second image frame corresponding to each of the N second moments, are used to determine whether the image acquisition process involves an injection attack behavior.

[0017] Considering that in the real image acquisition process (i.e., the image acquisition process that does not involve injection attack behavior), the real movement of the mobile terminal, i.e., the change in space, is strongly correlated with the movement reflected by the content of the video stream (i.e., the first image frame and the second image frames corresponding to the N second moments) acquired by its image acquisition device, i.e., the change in space. For example, the mobile terminal moves to the right relative to an object (e.g., a background object) in the video stream, and accordingly, the video stream will represent that the object moves to the left relative to the mobile terminal. For another example, the mobile terminal is close to an object in the video stream, and accordingly, the video stream will represent that the object is close to the mobile terminal. In view of this, in the above process, the first sensor data and the second sensor data corresponding to each of the N second moments, as well as the first image frame and the second image frame corresponding to each of the N second moments, are used to detect and identify whether the image acquisition process involves injection attack behavior. In addition, the above process is performed during the image acquisition process of the mobile terminal, which can realize the detection and identification of whether it involves injection attack behavior during the image acquisition process, and realize the detection and identification of injection attack behavior in business scenarios that rely on real-time video streams, so as to better protect the privacy and property safety of users. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0019] Figure 1A A schematic diagram of an implementation scenario of an embodiment disclosed in this specification;

[0020] Figure 1B A schematic diagram of an implementation framework of an embodiment disclosed in this specification;

[0021] Figure 2 A flow chart of a method for identifying injection attack behavior provided by an embodiment;

[0022] Figure 3A A schematic diagram of data acquisition conditions embodied by a time axis provided in an embodiment;

[0023] Figure 3B A schematic diagram of an injection attack behavior model provided by an embodiment;

[0024] Figure 4 A flow chart of a method for identifying injection attack behavior provided by an embodiment;

[0025] Figure 5 A schematic block diagram of a device for identifying injection attack behavior provided in an embodiment. DETAILED DESCRIPTION

[0026] The technical solutions of the embodiments of this specification will be described in detail below with reference to the accompanying drawings.

[0027] The embodiments of this specification disclose a method and device for identifying injection attack behaviors. The application scenarios and technical concepts of the method are first introduced as follows:

[0028] As mentioned above, due to the continuous development of AI-based content generation technology, its forged images (or videos) are becoming more and more realistic. In the scenario of real-time communication or identity authentication using real-time communication software, it is becoming increasingly difficult to identify and defend only from the image or video itself. In order to ensure the privacy information and / or property security of users in the above scenarios, it is crucial to provide a new method for identifying injection attack behaviors for defense.

[0029] In view of this, the inventor proposes a method for identifying injection attack behaviors, so as to better realize timely identification of injection attack behaviors in business scenarios that rely on real-time video streams. Figure 1A A schematic diagram of an implementation scenario according to an embodiment disclosed in this specification is shown. In this implementation scenario, a schematic diagram of the structure of a communication system is exemplarily shown, and the communication system is used to support communication services that rely on real-time video streaming and is applied to real-time communication scenarios, which can be instant video call scenarios or video conferencing scenarios.

[0030] The communication system may include a server and at least two mobile terminals, as shown in FIG. IA, wherein the at least two mobile terminals include: mobile terminal 1, mobile terminal 2, ..., mobile terminal M, where M is a positive integer. Each mobile terminal is installed with an application (hereinafter referred to as a communication application) for corresponding users to perform real-time communication, and the server can provide data transmission services in the real-time communication process for each mobile terminal (i.e., the communication application therein).

[0031] For example, during a video call, all mobile terminals start their image acquisition devices to capture images and transmit them to other mobile terminals participating in the instant video call process. At this time, the server can transmit the video stream obtained from mobile terminal 1 to mobile terminal 2-mobile terminal M, and transmit the video stream obtained from mobile terminal 2 to mobile terminal 1, and mobile terminal 3-mobile terminal M, and so on. For another example, during a video conference, the server can transmit the video stream obtained from the mobile terminal side of the shared screen to other mobile terminal sides. It can be understood that in some examples, other non-mobile terminals (not shown in the figure) may also be included in the communication system. In this implementation scenario, in order to better protect the privacy information and property safety of users, the communication system supports the recognition function of injection attack behavior to defend against injection attack behavior. Exemplarily, the recognition function of injection attack behavior can be implemented on each mobile terminal side or on the server side.

[0032] In the case where the injection attack behavior recognition function is implemented on each mobile terminal side, each mobile terminal (participating in real-time communication) can be installed with a target program, which is run by the processor of the mobile terminal to identify and detect whether the image acquisition process of the mobile terminal involves injection attack behavior during the image acquisition process (i.e., the process of acquiring video streams in real-time communication scenarios). Exemplarily, the target program can exist in the form of an SDK (Software Development Kit). In some cases, the target program can exist in the form of a plug-in of the aforementioned communication application, or in the form of an independent application. In the case of existing in the form of an independent application, the target program can be bound to the communication application to provide the communication application with the injection attack behavior recognition function.

[0033] In the case where the injection attack behavior recognition function is implemented on the server side, the server can install a target program, which is run by the processor of the server to obtain the sensor data and image frames corresponding to each moment in the image acquisition process of each mobile terminal (or mobile terminal sharing a screen) during the image acquisition process of the corresponding mobile terminal (that is, the process of acquiring a video stream in a real-time communication scenario), and combine the sensor data and image frames corresponding to each moment, and the sensor data and image frames corresponding to each moment before each moment (that is, combine the sensor data and image frames corresponding to multiple moments acquired within a decision cycle, N is a positive integer, and its value can be determined according to the length of a preset decision cycle), and jointly identify and detect whether its image acquisition process involves injection attack behavior.

[0034] It can be understood that the injection attack behavior recognition function is implemented on the mobile terminal side or on the server side, and the implementation process is similar. The following takes the implementation of the injection attack behavior recognition function on the mobile terminal side as an example, that is, taking the mobile terminal executing the injection attack behavior recognition process as an example, to explain its implementation process.

[0035] Among them, the process of each mobile terminal executing the identification process of injection attack behavior is similar. The following takes any mobile terminal (for example, mobile terminal 1, which needs to collect video streams to share with other mobile terminals in the real-time communication process) as an example to illustrate the implementation process of its execution of the identification process of injection attack behavior, that is, providing the identification function of injection attack behavior. The implementation process of other mobile terminals executing the identification process of injection attack behavior can refer to the implementation process of mobile terminal 1 executing the identification process of injection attack behavior.

[0036] like Figure 1B As shown, the mobile terminal 1 is provided with a sensor, and the sensor is used to sense the movement of the mobile terminal 1. The mobile terminal 1 is also provided with an image frame acquisition unit, and the image frame acquisition unit is implemented as a camera of the mobile terminal, for example.

[0037] Exemplarily, the mobile terminal 1 may be provided with various types of sensors, wherein the sensors may include, but are not limited to, acceleration sensors, gyroscopes, magnetic field sensors (such as compasses), etc. Figure 1B As shown, the acceleration sensor may include a linear acceleration sensor and / or an angular acceleration sensor.

[0038] During real-time communication, the image frame acquisition unit of the mobile terminal 1 can acquire image frames in real time (continuously acquired image frames constitute a video stream), and the sensor of the mobile terminal can sense the movement of the mobile terminal, that is, the change of its position in space, in real time.

[0039] During the image acquisition process of the mobile terminal 1, the target program installed in the mobile terminal 1 (running on the processor) can obtain data corresponding to each moment in the image acquisition process, and the data may include sensor data corresponding to the sensor in the mobile terminal 1, and image frames corresponding to the image frame acquisition unit of the mobile terminal 1; further, the identification process of the injection attack behavior can be executed for the acquired data corresponding to each moment in the image acquisition process.

[0040] like Figure 1BAs shown, specifically, during the image acquisition process of the mobile terminal 1, the target program can be run to determine whether the image acquisition process of the mobile terminal 1 involves injection attack behavior based on the image frames and sensor data corresponding to each moment in the decision cycle when the moment corresponding to each decision cycle arrives. Specifically, during the image acquisition process of the mobile terminal 1, when the moment corresponding to a certain decision cycle arrives, the first sensor data and the first image frame corresponding to the first moment in the image acquisition process are obtained; and the second sensor data and the second image frame corresponding to each of the N pre-stored second moments are obtained, wherein the N second moments are the first N moments of the first moment in the image acquisition process; then, the first sensor data and the first image frame, as well as the second sensor data and the second image frame corresponding to each of the N second moments, are used to determine whether the image acquisition process involves injection attack behavior. N is a positive integer, and its value can be determined according to the length of the preset decision cycle.

[0041] In the above process, the first sensor data and the second sensor data corresponding to each of the N second moments, as well as the first image frame and the second image frame corresponding to each of the N second moments, are used to detect and identify whether the image acquisition process involves injection attack behavior. In addition, the above process is performed during the image acquisition process of the mobile terminal, which can detect and identify whether it involves injection attack behavior during the image acquisition process, and detect and identify injection attack behavior in business scenarios that rely on real-time video streams, thereby better protecting the privacy and property safety of users.

[0042] Moreover, in actual scenarios, sensor data in the image acquisition process is often difficult to forge. Furthermore, it is relatively more difficult to forge sensor data and images that have a consistent relationship in the same image acquisition process, that is, matching. In view of this, the above process can better realize the accurate detection of whether the image acquisition process involves injection attack behavior, better improve the security of the corresponding business scenarios that rely on real-time video streams, and thus improve the privacy information and / or product security of the corresponding users.

[0043] In the above process, in some possible examples, if it is determined that the image acquisition process of the mobile terminal 1 involves injection attack behavior, the user account corresponding to the mobile terminal 1 (hereinafter referred to as the first user account) can be locked and determined as a user account with potential security risks. Then, exemplarily, the target program can (for example, notify the communication application) no longer transmit the acquired image frame corresponding to the image acquisition process to the server, so as to no longer transmit the image frame corresponding to its image acquisition process to the terminal corresponding to other user accounts (i.e., other mobile terminals) through the server; another example, the server can notify other user accounts that the image acquisition process of the first user account involves injection attack behavior, so as to remind other user accounts to prevent being deceived. Another example is that the real user corresponding to the first user account can be notified of abnormalities in the relevant account through other contact methods (such as email, mobile phone number, etc.) set by the user history corresponding to the first user account to help the real user avoid risks. Thereby, the privacy information and property safety of each user in the real-time communication process are protected.

[0044] If it is determined that the image acquisition process of mobile terminal 1 does not involve injection attack behavior, the image frame corresponding to the image acquisition process can continue to be transmitted to the server, so that the first image frame can be transmitted to the terminal corresponding to other user accounts (i.e., other mobile terminals) through the server without affecting the real-time call process.

[0045] When the injection attack recognition function is implemented on the server side, that is, the target program is installed on the server side, if it is determined that the image acquisition process of the mobile terminal 1 involves an injection attack, then after locking the first user account corresponding to the mobile terminal 1 and determining it as a user account with potential safety hazards, in order to ensure the privacy information and / or property safety of the real user corresponding to the first user account and the users corresponding to other user accounts that communicate with the first user account in real time, illustratively, the real user corresponding to the first user account can be notified of an abnormality in the relevant account through other contact methods set in the user history corresponding to the first user account to help the real user avoid risks; and illustratively, other user accounts that communicate with the first user account in real time can be locked, and the other user accounts can be notified that the image acquisition process of the first user account involves an injection attack to prevent users of other user accounts from being deceived. If it is determined that the image acquisition process of the mobile terminal 1 does not involve an injection attack, the image frames obtained from the mobile terminal 1 can continue to be transmitted to the terminals corresponding to other user accounts (i.e., other mobile terminals), without affecting the progress of their real-time call process.

[0046] It is understandable that the injection attack behavior identification process can also be applied to the identity authentication process in the identity authentication system to identify and detect whether the image acquisition process corresponding to the identity authentication process involves injection attack behavior. During the identity authentication process, if it is determined that the image acquisition process of the mobile terminal involves injection attack behavior, the identity authentication process of the corresponding user can be interrupted, and the corresponding user can be notified of abnormalities in his or her related account through other contact methods set by the corresponding user in history, so as to help the real user avoid risks.

[0047] The following is a detailed description of the method for identifying injection attack behaviors provided in this specification in conjunction with specific embodiments.

[0048] Figure 2 A flow chart of a method for identifying injection attack behavior in one embodiment of the present specification is shown. The method is executed by an electronic device, and the electronic device can be implemented by any device, equipment, platform, device cluster, etc. with computing and processing capabilities. In some examples, the electronic device can be a mobile terminal, and the mobile terminal can be provided with an image frame acquisition device (such as a camera, etc.) and a sensor for sensor data acquisition; the electronic device can also be a server that can communicate with the aforementioned mobile terminal and provide data storage and computing services for the mobile terminal. Exemplarily, the mobile terminal can be, but is not limited to: a mobile terminal device such as a mobile phone and a tablet computer.

[0049] Exemplarily, the sensor provided by the mobile terminal may include multiple types of first sensors for sensing the movement of the mobile terminal, and the first sensor may include, for example, but not limited to: the aforementioned acceleration sensor, gyroscope, magnetic field sensor, compass, etc., and the acceleration sensor may include a linear acceleration sensor and / or an angular acceleration sensor. In some other possible examples, the sensor provided by the mobile terminal may also include a second sensor for sensing the brightness of the environment in which the mobile terminal is located, and the second sensor may include, for example, but not limited to: the aforementioned photodiode, photoresistor diode, etc., which can sense ambient light. In this way, the sensor data corresponding to the first sensor and the sensor data corresponding to the second sensor, as well as the corresponding image frame corresponding to the image frame acquisition device (whose content can indicate both the movement of the mobile terminal during the image acquisition process and the ambient light condition of the mobile terminal during the image acquisition process) can be combined to determine whether the image acquisition process involves injection attack behavior, so as to realize the recognition of whether the image acquisition process involves injection attack behavior from multiple angles, and better improve the recognition accuracy.

[0050] The following is an introduction to the identification process of injection attack behavior. Figure 2 As shown, the injection attack behavior identification method includes the following steps S210-S230:

[0051] In step S210, during the image acquisition process of the mobile terminal, first sensor data and a first image frame corresponding to a first moment in the image acquisition process are obtained, wherein the first sensor data corresponds to a sensor in the mobile terminal, and the sensor is used to sense the movement of the mobile terminal.

[0052] The following takes the electronic device as a mobile terminal as an example to introduce the injection attack behavior identification method provided by this embodiment. When the electronic device is a server that can communicate with the mobile terminal, its implementation process is similar to the implementation process when the electronic device is a mobile terminal. Its implementation process can refer to the implementation process when the electronic device is a mobile terminal, and it will not be repeated here. Among them, when the electronic device is a server, the mobile terminal is required to send the sensor data and image frames corresponding to the image acquisition process to the server through the network, so that the server can perform the subsequent injection attack behavior identification process based on the acquired sensor data and image frames.

[0053] In the case where the electronic device is a mobile terminal, the mobile terminal may be installed with a target program, and the target program runs on a processor of the mobile terminal to execute a recognition process of injection attack behavior.

[0054] In some exemplary business scenarios, such as real-time communication scenarios (for example, including video call scenarios and video conferencing scenarios) and identity authentication scenarios, after the application of the corresponding business scenario is triggered, the application can call the image frame acquisition device of the mobile terminal to start image acquisition, and the corresponding mobile terminal is in the image acquisition process. In addition, the application can call the target program, and accordingly, the target program is called to execute the identification process of the injection attack behavior for the image acquisition process of the mobile terminal.

[0055] Exemplarily, the target program can obtain the sensor data and image frames corresponding to the mobile terminal during the image acquisition process in real time. The sensor data corresponding to the image acquisition process may be the sensor data actually acquired by the sensor of the mobile terminal during the image acquisition process, or may be the sensor data forged by the black market of the mobile terminal corresponding to the sensor of the mobile terminal. The image frames corresponding to the image acquisition process may be the image frames actually acquired by the image frame acquisition device of the mobile terminal, or may be forged image frames injected by the black market of the mobile terminal.

[0056] After the target program obtains the corresponding sensor data and image frames in the image acquisition process, it can store the obtained sensor data and image frames in the designated storage space for identification and detection of whether the image acquisition process involves injection attack behavior. In some examples, considering that in some exemplary scenarios, some foreground objects in the image frame may be in a moving state, when the mobile terminal moves and some foreground objects also move, the movement of the mobile terminal indicated in the image frame, that is, the change in spatial position, may be biased. In order to better ensure the accuracy of the result of whether the image acquisition process involves injection attack behavior, after each image frame corresponding to the image acquisition process is obtained, the target program can identify the foreground object in the image frame and remove the foreground object from the image frame to obtain a background frame corresponding to the image frame that only includes background information. After that, the obtained sensor data and the background frame corresponding to the image frame are stored in the designated storage space for identification and detection of whether the image acquisition process involves injection attack behavior.

[0057] In some examples, the target program can periodically trigger the recognition process of injection attack behavior for the image acquisition process of the mobile terminal every time the moment corresponding to the decision cycle is detected; or it can also trigger the recognition process of injection attack behavior for the image acquisition process of the mobile terminal after detecting the recognition process triggering instruction. The following takes the target program periodically triggering the recognition process of injection attack behavior for the image acquisition process of the mobile terminal as an example for explanation.

[0058] The length of the decision cycle can be set according to actual needs. Assuming that the decision cycle or the decision time window is set to ΔT, the time of the first decision is T0. Accordingly, when the target program detects the arrival of the time XΔT+T0, it can be understood that the time T corresponding to the decision cycle has arrived. X can be 0 or a positive integer.

[0059] When the target program detects the arrival of the moment T corresponding to the decision cycle, it obtains the first sensor data and the first image frame corresponding to the first moment in the image acquisition process.

[0060] Among them, the first moment can be the moment T corresponding to the decision cycle; it can also be the image frame acquisition moment closest to the moment T corresponding to the decision cycle; it can also be the image frame acquisition moment second closest to the moment T corresponding to the decision cycle; it can also be the image frame acquisition moment third closest to the moment T corresponding to the decision cycle, and so on.

[0061] Accordingly, the N second moments mentioned later may be the N image frame acquisition moments before the first moment in the decision cycle, wherein the value of N may be determined according to the decision cycle.

[0062] The first image frame corresponding to the first moment may refer to the image frame collected at the first moment, and the first sensor data corresponding to the first moment may refer to the sensor data acquired within a preset collection window corresponding to the first moment.

[0063] In some specific examples, when the first moment is the moment T corresponding to the decision cycle, the first sensor data corresponding to the first moment may refer to the sensor data acquired between the second moment before the first moment and the first moment. Specifically, it can be determined according to the timestamp information carried by the sensor data mentioned later.

[0064] In some specific examples, when the first moment is the image frame acquisition moment closest to the moment T corresponding to the decision cycle, the first sensor data corresponding to the first moment may refer to the sensor data acquired between the second moment before the first moment and the first moment, and the sensor data acquired between the first moment and the moment T corresponding to the decision cycle. Figure 3A shown.

[0065] In some implementations, when the first moment is the moment T corresponding to the decision cycle, the first image frame corresponding to the first moment may be an image frame acquired in real time. When the first moment is the image frame acquisition moment closest to the moment T corresponding to the decision cycle, the first image frame corresponding to the first moment may be an image frame acquired in real time (for example, the moment T corresponding to the decision cycle is shorter than the first moment); or it may be an image frame acquired from a designated storage space (only the background information therein may be retained), for example, the moment T corresponding to the decision cycle is longer than the first moment.

[0066] In some possible examples, when the sensor of the mobile terminal includes an acceleration sensor, a gyroscope, and a magnetic field sensor, the first sensor data may include, for example: an acceleration vector corresponding to the acceleration sensor, an angular velocity vector corresponding to the gyroscope, and a magnetic field intensity vector corresponding to the magnetic field sensor (such as Figure 1B Correspondingly, the second sensor data corresponding to the subsequent N second moments may also include an acceleration vector corresponding to the acceleration sensor, an angular velocity vector corresponding to the gyroscope, and a magnetic field intensity vector corresponding to the magnetic field sensor.

[0067] Among them, the acceleration vector is used to indicate the acceleration component of the mobile terminal in each coordinate system (including the XYZ axis) under the specified three-dimensional rectangular coordinate space; the angular velocity vector is used to indicate the angular velocity component of the mobile terminal in each coordinate system (including the XYZ axis) under the specified three-dimensional rectangular coordinate space, and is used to indicate the magnetic field intensity component of the mobile terminal in each coordinate system (including the XYZ axis) under the specified three-dimensional rectangular coordinate space.

[0068] It can be understood that in a real image acquisition process (i.e., an image acquisition process that does not involve injection attack behavior), the first sensor data can be sensor data actually collected by the sensor of the mobile terminal, and the first image frame is an image frame actually collected by the image frame acquisition device of the mobile terminal.

[0069] In the image acquisition process involving injection attack, the mobile terminal is usually placed in a certain place, that is, it will not be moved by the user, to collect images. At this time, the black industry (such as a program installed on the mobile terminal) will replace the image frames actually collected by the image frame acquisition device of the mobile terminal with its forged image frames, and then transmit the forged image frames to the target program to achieve the injection attack. At this time, the target program obtains the image frames forged by the black industry.

[0070] In the above-mentioned image acquisition process involving injection attack behavior, the sensor in the mobile terminal can sense the movement of the mobile terminal and collect corresponding sensor data. After that, the sensor transmits the collected sensor data to the target device. In a subsequent situation, the black industry does not tamper with the sensor data transmitted by the sensor to the target program. Accordingly, the target program can obtain the sensor data actually collected by the sensor, and the sensor data at this time combined with the sensor data collected at the previous moment may indicate that the mobile terminal is in a stationary state. In another subsequent situation, the black industry will replace the sensor data actually collected by the sensor of the mobile terminal with its forged sensor data, thereby transmitting the forged sensor data to the target program to achieve the injection attack. At this time, the target program obtains the sensor data forged by the black industry.

[0071] Accordingly, during the image acquisition process of the mobile terminal, the first sensor data corresponding to the first moment in the image acquisition process may be sensor data actually acquired by the sensor of the mobile terminal, or sensor data corresponding to the sensor of the mobile terminal forged by the black industry; and the first image frame corresponding to the first moment in the image acquisition process may be an image frame actually acquired by the image frame acquisition device of the mobile terminal, or an image frame corresponding to the image frame acquisition device of the mobile terminal forged by the black industry. The image frame may be forged by the content generation technology based on AI technology, or may be forged based on other image (video) modification technologies.

[0072] In some examples, the first sensor data and the first image frame may also carry corresponding timestamp information. Exemplarily, in the case where the first sensor data is sensor data actually collected by the sensor of the mobile terminal, the timestamp information carried by the first sensor data may be the timestamp information when the sensor actually collects the sensor data; in the case where the first image frame is an image frame actually collected by the image frame acquisition device of the mobile terminal, the timestamp information carried by the first image frame may be the timestamp information when the image frame acquisition device actually collects the image frame. Another exemplary case is that the first sensor data is sensor data corresponding to the sensor of the mobile terminal forged by the black industry, the timestamp information carried by the first sensor data may be the timestamp information forged by the black industry based on the timestamp information corresponding to the real sensor data it replaces; in the case where the first image frame is an image frame forged by the black industry corresponding to the image frame acquisition device of the mobile terminal, the timestamp information carried by the first image frame may be the timestamp information forged by the black industry based on the timestamp information corresponding to the real image frame it replaces.

[0073] After the target program obtains the first sensor data and the first image frame corresponding to the first moment, in step S220, the second sensor data and the second image frame corresponding to each of the N pre-stored second moments are obtained, wherein the N second moments are the N moments before the first moment in the aforementioned image acquisition process.

[0074] It can be understood that the target program can store the acquired sensor data and image frames corresponding to each image acquisition process in a designated storage space, so as to combine the sensor data and image frames corresponding to the image acquisition process acquired subsequently to determine whether the image acquisition process involves injection attack behavior. Accordingly, in this step, after the target program obtains the first sensor data and the first image frame corresponding to the first moment, it can obtain the second sensor data and the second image frame corresponding to each of the N pre-stored second moments, i.e., the N moments before the first moment in the image acquisition process, from the designated storage space.

[0075] In some possible examples, as described above, the N second moments may be the N image frame acquisition moments before the first moment in the decision cycle. Accordingly, the second image frame corresponding to each second moment may be: the image frame acquired at the second moment. The second sensor data corresponding to each second moment may be the sensor data acquired within the preset acquisition window corresponding to the second moment. In a specific example, the second sensor data corresponding to each second moment may be the sensor data acquired between the second moment before the second moment and the second moment.

[0076] In some possible scenarios, the Nth moment before the first moment ( Figure 3AThe moment before the second moment t14) shown in FIG. Figure 3A The second moment t15) shown in FIG. 1 does not belong to the decision cycle. At this time, the Nth moment before the first moment ( Figure 3A The second sensor data corresponding to the second time t14) shown in FIG. 14 may be the earliest time corresponding to the decision cycle (eg, Figure 3A T0-ΔT) shown in , to the Nth moment before the first moment (such as Figure 3A The sensor data acquired between the second time t14) shown in FIG.

[0077] like Figure 3A As shown. Among them, the decision cycle is ΔT, the moment corresponding to the decision cycle is T0, the first moment t11 ​​is the image frame acquisition moment closest to the moment T0 corresponding to the decision cycle, and sensor data is also acquired between the first moment t11 ​​and the moment T0 corresponding to the decision cycle, which corresponds to the sensor data acquisition moment t21. The decision cycle ΔT also includes the first three second moments (image frame acquisition moments) of the first moment t11, which are t12, t13 and t14 (the corresponding image frames need to be acquired). Between the first moment t11 ​​and the second moment t12, there are multiple sensor data acquisition moments t2a (the corresponding sensor data need to be acquired); between the second moment t12 and the second moment t13, between the second moment t13 and the second moment t14, and between the second moment t14 and T0-ΔT, there are multiple sensor data acquisition moments t2b (the corresponding sensor data need to be acquired).

[0078] That is to say, every time the target program detects the arrival of a moment corresponding to a decision cycle, it can obtain all image frames (which may be image frames truly captured by the image capture device or forged image frames injected by the black industry) transmitted from the image acquisition device of the mobile terminal to the target program within the upcoming decision cycle (i.e., the current decision cycle), and obtain all sensor data (which may be sensor data truly captured by the sensor or forged sensor data injected by the black industry) transmitted from the target program from the sensor of the mobile terminal within the upcoming decision cycle (i.e., the current decision cycle).

[0079] After obtaining the first sensor data and the first image frame corresponding to the first moment and the second sensor data and the second image frame corresponding to N second moments, in step S230, using the aforementioned first sensor data and the first image frame, and the aforementioned second sensor data and the second image frame corresponding to the N second moments, it is determined whether the image acquisition process involves an injection attack behavior.

[0080] Considering that in the actual image acquisition process, the movement of the mobile terminal in the image acquisition process indicated by the content in the image frame actually captured by the image frame acquisition device acquired by the target program matches the movement of the mobile terminal in the image acquisition process, that is, the movement of the mobile terminal in the image acquisition process indicated by the sensor data actually captured by the sensor of the mobile terminal. That is, in the actual image acquisition process, there is a strong correlation between the acquired image frames and sensor data, and there is consistency.

[0081] Moreover, in the actual image acquisition process, the changes in the environmental conditions (such as ambient light conditions) of the mobile terminal can also be represented between image frames, and the data about the ambient light of the mobile terminal actually collected by the sensor of the mobile terminal (such as the second sensor) can also represent the changes in the environmental conditions (such as ambient light conditions) of the mobile terminal, and the changes represented by the two are matched and consistent.

[0082] In the image acquisition process involving injection attack behavior, one implementation is that the image frame corresponding to the image acquisition process acquired by the target program is forged by the black industry, and the sensor data corresponding to the image acquisition process acquired by the target program is based on the data actually acquired by the sensor of the mobile terminal, which can indicate the real movement of the mobile terminal. At this time, the movement of the mobile terminal indicated by the content of each of the first image frame and the second image frames corresponding to the N second moments is less likely to match the movement of the mobile terminal indicated by the sensor data. Accordingly, in the image acquisition process involving injection attack behavior, the correlation between the acquired image frames and the sensor data will be weaker, and the possibility of consistency is smaller.

[0083] In addition, when the sensor of the mobile terminal is also used to sense the ambient light of the environment in which the mobile terminal is located, the possibility of matching the changes in the ambient light of the mobile terminal as indicated by the contents of the first image frame and each of the second image frames corresponding to the N second moments with the actual ambient light conditions of the mobile terminal as indicated by the sensor data actually collected by the sensor of the mobile terminal is also low.

[0084] In another implementation of the image acquisition process involving injection attack behavior, the image frame corresponding to the image acquisition process acquired by the target program is forged by the image content generated by the black industry based on AI technology or other image modification technology, and the sensor data corresponding to the image acquisition process acquired by the target program is also forged by the black industry. It is difficult to forge sensor data, and it is even more difficult to forge sensor data that matches the motion indicated by the image frame forged by the black industry corresponding to the acquired image acquisition process. Accordingly, in the image acquisition process involving injection attack behavior, the correlation between the acquired image frame and the sensor data will be weak.

[0085] In view of the above situation, the target program can use the first sensor data and the first image frame, and the second sensor data and the second image frame corresponding to each of the N second moments to determine whether the image acquisition process involves injection attack behavior.

[0086] Taking into account that in some exemplary scenarios, some foreground objects in the image frame may be in a moving state. When the mobile terminal moves and some foreground objects also move, the movement of the mobile terminal indicated in the image frame may deviate. In order to better ensure the accuracy of the results of determining whether the image acquisition process involves injection attack behavior, the background information in the image frame can be used in combination with the acquired sensor data to determine whether the image acquisition process involves injection attack behavior.

[0087] In view of the above situation, in some possible examples, after acquiring the image frame, the target program can remove the foreground object in the image frame to obtain a background frame that only retains the background information, so as to facilitate the identification and detection of whether the image acquisition process involves injection attack behavior. The process of removing the foreground object in the image frame is introduced below.

[0088] Assuming that the first moment is the moment T corresponding to the decision cycle, the second image frames corresponding to the N second moments acquired in step S220 are the image frames (including background information) from which the foreground object is removed; accordingly, step S230 may include the following steps 11-12:

[0089] In step 11, the foreground object in the first image frame is removed to obtain a first background frame corresponding to the first image frame. In this step, the target program can also obtain a depth map corresponding to the first image frame, wherein the depth map includes depth values ​​corresponding to each pixel in the first image frame (i.e., depth estimates in the subsequent first depth map). Accordingly, the foreground object in the first image frame can be removed based on the depth map corresponding to the first image frame to obtain a first background frame corresponding to the first image frame.

[0090] Exemplarily, pixel points in the first image frame whose corresponding depth values ​​are less than a preset depth threshold may be determined as pixel points involved in a foreground object, and the pixel values ​​of such pixel points may be set to a first value (for example, 0); pixel points in the first image frame whose corresponding depth values ​​are not less than the preset depth threshold may be determined as background pixel points, and the pixel values ​​of such pixel points may be retained, thereby removing the foreground object in the first image frame and obtaining a first background frame corresponding to the first image frame.

[0091] In some possible examples, the camera of the mobile terminal may include a depth camera and an RGB camera, wherein the RGB camera is used to capture a color image of the corresponding scene, i.e., to obtain the aforementioned first image frame, and the depth camera is used to capture the depth value of the scene, and a depth map corresponding to the aforementioned first image frame can be obtained. That is to say, in this example, the depth map corresponding to the first image frame can be provided by the depth camera of the mobile terminal.

[0092] In some other possible examples, the depth map corresponding to the first image frame may be determined based on a trained deep learning model. Accordingly, step 11 may include the following steps 111-112:

[0093] In step 111, a first depth map is obtained based on the first image frame through a trained depth estimation model, wherein the first depth map includes each depth estimation value corresponding to each pixel in the first image frame.

[0094] The depth estimation model may be a model that has been trained based on each sample image and its corresponding label data indicating a sample depth value corresponding to each pixel in the sample image.

[0095] In this step, the first image frame may be input into a trained depth estimation model to process the first image frame through the trained depth estimation model to obtain a first depth map including depth estimation values ​​corresponding to each pixel in the first image frame.

[0096] Thereafter, in step 112, the foreground object in the first image frame is removed using the first depth map and a preset depth threshold to obtain a first background frame.

[0097] Specifically, the target program can compare each depth estimate in the first depth map with a preset depth threshold, and generate a foreground mask based on the comparison result, wherein the size of the foreground mask can be the same as the size of the first depth map, that is, the same as the size of the first image frame, and each point in the foreground mask has a one-to-one correspondence with each pixel in the first depth map, and each point in the foreground mask has a one-to-one correspondence with each pixel in the first image frame.

[0098] If the depth estimation value of the pixel point in the i-th row and j-th column in the first depth map is less than the preset depth threshold, it can be determined that the pixel point corresponds to the foreground object, and accordingly, the value of the pixel point in the i-th row and j-th column of the foreground mask can be set to the first value (for example, 0); if the depth estimation value of the pixel point in the i-th row and j-th column in the first depth map is not less than the preset depth threshold, it can be determined that the pixel point corresponds to the background, and accordingly, the value of the pixel point in the i-th row and j-th column of the foreground mask can be set to the second value (for example, 1). Afterwards, the foreground object in the first image frame is removed using the foreground mask to obtain the first background frame. Among them, for the pixel point in the first image frame corresponding to the point whose value in the foreground mask is the first value, its pixel value is set to the first value (for example, 0); for the pixel point in the first image frame corresponding to the point whose value in the foreground mask is the second value, its pixel value is retained, so as to remove the foreground object in the first image frame using the foreground mask, thereby obtaining the first background frame.

[0099] After obtaining the first background frame corresponding to the first image frame, in step 12, the first sensor data and the first background frame, as well as the second sensor data and the second image frames corresponding to the N second moments, are used to determine whether the image acquisition process involves injection attack behavior. The second image frames corresponding to the N second moments are image frames from which the foreground object is removed.

[0100] Through the above method, real-time deep analysis is used to extract background information in image frames (i.e., foreground objects are removed from image frames). The spatial changes of the background information in image frames in the image frame sequence can more accurately indicate the spatial position changes, i.e., the movement of the mobile terminal. The background information in the image frames can better improve the accuracy of injection attack behavior recognition.

[0101] It can be understood that the second image frames with the foreground objects removed and stored in the designated storage space are all removed in the aforementioned manner.

[0102] In some possible examples, thereafter, whether the image acquisition process involves injection attack behavior can be determined by utilizing the first movement of the mobile terminal indicated by the first sensor data and the second sensor data corresponding to each of the N second moments, as well as the consistency between the second movement of the mobile terminal indicated by the aforementioned first background frame (or first image frame) and the second image frame corresponding to each of the N second moments.

[0103] The following uses the first background frame as an example to introduce the process of determining whether the image acquisition process involves injection attack behavior. For the process of determining whether the image acquisition process involves injection attack behavior using image frames from which the foreground objects have not been removed, please refer to the process of determining whether the image acquisition process involves injection attack behavior using the first background frame, which will not be repeated here.

[0104] After acquiring the first background frame, the target program can extract each feature point from the first background frame based on the specified feature extraction algorithm, and obtain each feature point in the second image frame corresponding to each of the N second moments; then use Brute-Force Matcher or FLANN (Fast Library for ApproximateNearest Neighbors) based fast approximate nearest neighbor search to match multiple feature point pairs of the nth image frame and the n-1th image frame in the first background frame and the second image frames corresponding to each of the N second moments. Then, the RANSAC (random sample consensus) algorithm or other possible screening algorithms can be used to eliminate feature point pairs that may be abnormal. Then, based on the filtered feature point pairs, the essential matrix corresponding to the nth image frame and the n-1th image frame is calculated, and the image pose data of the nth image frame relative to the n-1th image frame is obtained by decomposing the essential matrix, that is, the image pose data corresponding to the nth image frame is obtained. By analogy, the image pose data corresponding to the first moment and the image pose data corresponding to each of the N second moments are obtained. Among them, each feature point in the second image frame corresponding to each of the N second moments is stored in a designated storage space, so as to better improve the efficiency of determining whether the image acquisition process involves injection attack behavior.

[0105] The aforementioned specified feature extraction algorithm may be, but is not limited to, a SIFT (Scale-Invariant Feature Transform) algorithm, an optical flow tracking algorithm, an ORB (Oriented FAST and Rotated BRIEF) or a speeded up robust features (SURF) or other feature extraction algorithms.

[0106] Next, the target program can determine the first motion condition of the mobile terminal indicated by it based on the image posture data corresponding to the first moment and the image posture data corresponding to each of the N second moments, and determine the second motion condition of the mobile terminal indicated by it based on the first sensor data and the second sensor data corresponding to each of the N second moments, and then determine whether the image acquisition process involves injection attack behavior based on the first motion condition and the second motion condition. If it is determined that the first motion condition and the second motion condition match, it is determined that the image acquisition process does not involve injection attack behavior; conversely, if it is determined that the first motion condition and the second motion condition do not match, it is determined that the image acquisition process involves injection attack behavior.

[0107] In some other possible examples, it is also possible to determine whether the image acquisition process involves injection attack behavior by combining the first sensor data and the second sensor data corresponding to the N second moments, and the first image frame and the second image frame corresponding to the N second moments through a trained deep learning-based network model. Specifically, step S230 may include the following step 21:

[0108] In step 21, using the first sensor data and the second sensor data corresponding to each of the N second moments, and the first image frame and the second image frame corresponding to each of the N second moments, a trained injection attack behavior recognition model is used to determine whether the image acquisition process involves an injection attack behavior.

[0109] Among them, the injection attack behavior identification model is a model that is pre-trained based on the sample data corresponding to each sample collection process and its corresponding label data. The sample data includes sample sensor data and sample image data corresponding to the corresponding sample collection process. The label data is used to indicate whether the corresponding sample collection process involves injection attack behavior.

[0110] In this step, the first sensor data and the second sensor data corresponding to each of the N second moments are combined into a sensor data sequence in the order of the corresponding moments; the first image frame and the second image frames corresponding to each of the N second moments are combined into an image frame sequence; the sensor data sequence and the image frame sequence are input into the trained injection attack behavior recognition model to process the sensor data sequence and the image frame sequence through the injection attack behavior recognition model to determine whether the image acquisition process involves injection attack behavior.

[0111] Exemplarily, when the first sensor data includes sensor data corresponding to multiple different sensor data collection moments, the sensor data corresponding to the multiple different sensor data collection moments in the first sensor data are sorted in the order of the corresponding sensor data collection moments. Similarly, the sensor data corresponding to multiple different sensor data collection moments in the second sensor data corresponding to each second moment are sorted in the order of the corresponding sensor data collection moments. In other words, the sensor data sequence includes sensor data sorted in the order of the corresponding sensor data collection moments.

[0112] In some other possible examples, the first sensor data and the second sensor data corresponding to each of the N second moments, as well as the first background frame corresponding to the first image frame and the second image frames corresponding to each of the N second moments, can be used to determine whether the image acquisition process involves injection attack behavior through a trained injection attack behavior recognition model. The second image frames corresponding to each of the N second moments are image frames from which foreground objects are removed.

[0113] In some possible implementations, such as Figure 3B As shown, the injection attack behavior recognition model may include a first encoder, a second encoder, and a classification network; illustratively, the first encoder and the second encoder may be implemented by a recurrent neural network, or by a network based on a transformer structure. The classification network may be implemented as a fully connected layer or other types of classifiers.

[0114] Accordingly, in step 21, the following steps 211-213 may be included:

[0115] In step 211, the first sensor data and the second sensor data corresponding to each of the N second moments are used to obtain a first feature sequence through a first encoder. In this step, a sensor data sequence consisting of the first sensor data and the second sensor data corresponding to each of the N second moments is input into the first encoder, so that the sensor data sequence is processed by the first encoder to obtain a first feature sequence.

[0116] In step 212, the first image frame and the second image frames corresponding to the N second moments are used to obtain a second feature sequence through a second encoder. In this step, an image frame sequence consisting of the first image frame and the second image frames corresponding to the N second moments is input into the second encoder to process the image frame sequence through the image frame sequence to obtain a second feature sequence.

[0117] Then, in step 213, the first feature sequence and the second feature sequence are used to determine whether the image acquisition process involves injection attack behavior through a classification network. In this step, the first feature sequence and the second feature sequence can be spliced ​​to obtain a spliced ​​feature sequence, which is input into the classification network and processed by the classification network to determine whether the image acquisition process involves injection attack behavior.

[0118] In some possible examples, the classification network may input a probability value indicating whether the image acquisition process involves an injection attack behavior. If the probability value is greater than a specified threshold, it may be determined that there is spatial consistency between the sensor data sequence and the image frame sequence, and accordingly, it is determined that the image acquisition process does not involve an injection attack behavior. If the probability value is not greater than the specified threshold, it may be determined that there is no spatial consistency between the sensor data sequence and the image frame sequence, and accordingly, it is determined that the image acquisition process involves an injection attack behavior.

[0119] Using an injection attack behavior recognition model that can process multimodal data, namely sensor data sequences and image frame sequences, the features of sensor data sequences and image frame sequences can be extracted respectively, thereby achieving accurate judgment on the spatial consistency of sensor data sequences and image frame sequences, and realizing real-time recognition and detection of whether the image acquisition process involves injection attack behavior, thereby improving detection accuracy.

[0120] In this embodiment, the first sensor data and the second sensor data corresponding to each of the N second moments, as well as the first image frame and the second image frame corresponding to each of the N second moments, are used to detect and identify whether the image acquisition process involves injection attack behavior. In addition, the above process is performed during the image acquisition process of the mobile terminal, which can detect and identify whether it involves injection attack behavior during the image acquisition process, and detect and identify injection attack behavior in business scenarios that rely on real-time video streams, thereby better protecting the privacy and property safety of users.

[0121] In the above process, the detection accuracy is much higher due to the use of spatial consistency reflected by sequence data of two different dimensions instead of just using the features of the image itself. In addition, the detection scope is not limited to the content generated by AIGC, but all injected video streams.

[0122] In some possible implementations, such as Figure 4 As shown, the injection attack behavior identification method may include the following steps S410-S440:

[0123] In step S410, during the image acquisition process of the mobile terminal, first sensor data and a first image frame corresponding to the first moment in the image acquisition process are obtained, wherein the first sensor data corresponds to a sensor in the mobile terminal, and the sensor is used to sense the movement of the mobile terminal. The implementation principle of step S410 is similar to the implementation principle of the aforementioned step S210, and its implementation process can refer to the implementation process of the aforementioned step S210, which will not be described in detail here.

[0124] In step S420, the first background frame and the first sensor data corresponding to the first moment are stored in the designated storage space, wherein the first background frame is obtained after the foreground object in the first image frame is removed. In this step, the foreground object is determined from the first image frame, and the foreground object is removed to obtain the first background frame, and then the first background frame and the first sensor data corresponding to the first moment are stored in the designated storage space, so as to trigger the timely identification and detection of whether the injection attack behavior is involved in the image acquisition process at a moment after the first moment.

[0125] In step S430, the second sensor data and the second image frame corresponding to each of the N pre-stored second moments are obtained, wherein the N second moments are the N moments before the first moment in the image acquisition process. In this step, the second sensor data and the second image frame corresponding to each of the N pre-stored second moments can be obtained from the aforementioned designated storage space. The second image frame is an image frame in which the foreground object is removed.

[0126] In step S440, it is determined whether the image acquisition process involves injection attack behavior by using the first sensor data and the first image frame, and the second sensor data and the second image frames corresponding to each of the N second moments.

[0127] Among them, the implementation principle of steps S430-S440 is similar to the implementation principle of the aforementioned steps S220-S230. The implementation process thereof can refer to the implementation process of the aforementioned steps S220-S230, and will not be repeated here.

[0128] Understandably, Figure 4 In the process shown, step S420 is executed first, and then steps S430-S440 are executed. In some other examples, steps S430-S440 may be executed first, and then step S420; or step S420 may be executed between steps S430 and S440, etc.

[0129] In the above process, an external storage, i.e., a designated storage space, is used to cache historical data, i.e., the second sensor data and the second image frame corresponding to each of the N second moments, and after obtaining the first sensor data and the first image frame corresponding to the first moment, the first background frame corresponding to the first sensor data and the first image frame is stored therein for the recognition process at the next moment after the first moment, so as to better realize the real-time and timely recognition and detection of the injection attack behavior in the image acquisition process, and better improve the recognition and detection efficiency.

[0130] The foregoing describes certain embodiments of the present specification, and other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims may be performed in an order different from that in the embodiments, and the desired results may still be achieved. In addition, the processes depicted in the accompanying drawings do not necessarily have to be performed in the specific order or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0131] Corresponding to the above method embodiment, the present specification embodiment provides an injection attack behavior identification device 500, whose schematic block diagram is as follows: Figure 5 As shown, including:

[0132] A first acquisition module 510 is configured to acquire, during an image acquisition process of a mobile terminal, first sensor data and a first image frame corresponding to a first moment in the image acquisition process, wherein the first sensor data corresponds to a sensor in the mobile terminal, and the sensor is used to sense a movement of the mobile terminal;

[0133] A second acquisition module 520 is configured to acquire second sensor data and second image frames corresponding to N pre-stored second moments, respectively, wherein the N second moments are N moments before the first moment in the image acquisition process;

[0134] The determination module 530 is configured to determine whether the image acquisition process involves injection attack behavior by using the first sensor data and the first image frame, and the second sensor data and the second image frame corresponding to each of the N second moments.

[0135] In some possible implementations, the second image frames corresponding to each of the N second moments are image frames from which foreground objects are removed; the determination module 530 is specifically configured to remove the foreground objects in the first image frame to obtain a first background frame corresponding to the first image frame; and determine whether the image acquisition process involves an injection attack by using the first sensor data and the first background frame, as well as the second sensor data and the second image frames corresponding to each of the N second moments.

[0136] In some possible implementations, the determination module 530 is specifically configured to obtain a first depth map based on the first image frame through a trained depth estimation model, wherein the first depth map includes each depth estimation value corresponding to each pixel point in the first image frame;

[0137] The foreground object in the first image frame is removed using the first depth map and a preset depth threshold to obtain the first background frame.

[0138] In some possible implementations, determining whether the image acquisition process involves an injection attack behavior includes:

[0139] By using the first sensor data and the second sensor data corresponding to each of the N second moments, and the first image frame and the second image frame corresponding to each of the N second moments, it is determined whether the image acquisition process involves injection attack behavior through a trained injection attack behavior recognition model.

[0140] In some possible implementations, the injection attack behavior recognition model includes a first encoder, a second encoder, and a classification network; the determination module 530 includes:

[0141] A first obtaining unit (not shown in the figure) is configured to obtain a first feature sequence by using the first sensor data and the second sensor data corresponding to each of the N second moments through the first encoder;

[0142] A second obtaining unit (not shown in the figure) is configured to obtain a second feature sequence by using the first image frame and the second image frames corresponding to the N second moments through the second encoder;

[0143] A determination unit (not shown in the figure) is configured to use the first feature sequence and the second feature sequence to determine whether the image acquisition process involves injection attack behavior through the classification network.

[0144] In some possible implementations, the first encoder and the second encoder are implemented by a recurrent neural network, or by a network based on a transformer structure.

[0145] In some possible implementations, it also includes: a storage module (not shown in the figure), configured to store the first background frame corresponding to the latest first moment and the first sensor data in a designated storage space after obtaining the first sensor data and the first image frame corresponding to the first moment in the image acquisition process, wherein the first background frame is obtained after the foreground object in the first image frame is removed.

[0146] The above device embodiments correspond to the method embodiments. For specific descriptions, please refer to the description of the method embodiments, which will not be repeated here. The device embodiments are obtained based on the corresponding method embodiments and have the same technical effects as the corresponding method embodiments. For specific descriptions, please refer to the corresponding method embodiments.

[0147] The embodiments of the present specification also provide a computer-readable storage medium on which a computer program is stored. When the computer program is executed in a computer, the computer is caused to execute the injection attack behavior identification method provided in the present specification.

[0148] The embodiment of the present specification also provides a computing device, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method for identifying injection attack behavior provided in the present specification is implemented.

[0149] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the storage medium and computing device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0150] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the embodiments of the present invention may be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions may be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.

[0151] The specific implementation methods described above further describe the purpose, technical solutions and beneficial effects of the embodiments of the present invention in detail. It should be understood that the above description is only a specific implementation method of the embodiments of the present invention and is not intended to limit the scope of protection of the present invention. Any modification, equivalent replacement, improvement, etc. made on the basis of the technical solution of the present invention shall be included in the scope of protection of the present invention.

Claims

1. A method for identifying injection attack behavior, comprising: During an image acquisition process of a mobile terminal, first sensor data and a first image frame corresponding to a first moment in the image acquisition process are acquired, wherein the first sensor data corresponds to a sensor in the mobile terminal, and the sensor is used to sense a movement of the mobile terminal; Acquire second sensor data and second image frames corresponding to respective N pre-stored second moments, wherein the N second moments are N moments before the first moment in the image acquisition process; It is determined whether the image acquisition process involves injection attack behavior by using the first sensor data and the first image frame, and the second sensor data and the second image frame corresponding to each of the N second moments.

2. The method of claim 1, wherein: The second image frames corresponding to the N second moments are image frames from which the foreground object is removed; The determining whether the image acquisition process involves injection attack behavior includes: Removing the foreground object from the first image frame to obtain a first background frame corresponding to the first image frame; Using the first sensor data and the first background frame, as well as the second sensor data and the second image frames corresponding to the N second moments, it is determined whether the image acquisition process involves injection attack behavior.

3. The method of claim 2, wherein: The removing the foreground object from the first image frame to obtain a first background frame corresponding to the first image frame includes: Based on the first image frame, obtaining a first depth map through a trained depth estimation model, wherein the first depth map includes each depth estimation value corresponding to each pixel in the first image frame; The foreground object in the first image frame is removed by using the first depth map and a preset depth threshold to obtain the first background frame.

4. The method of claim 1, wherein: The determining whether the image acquisition process involves injection attack behavior includes: By using the first sensor data and the second sensor data corresponding to each of the N second moments, and the first image frame and the second image frame corresponding to each of the N second moments, it is determined whether the image acquisition process involves injection attack behavior through a trained injection attack behavior recognition model.

5. The method of claim 4, wherein: The injection attack behavior recognition model includes a first encoder, a second encoder and a classification network; The determining whether the image acquisition process involves injection attack behavior includes: Using the first sensor data and the second sensor data corresponding to each of the N second moments, through the first encoder, a first feature sequence is obtained; Using the first image frame and the second image frames corresponding to the N second moments, respectively, through the second encoder, a second feature sequence is obtained; Using the first feature sequence and the second feature sequence, through the classification network, it is determined whether the image acquisition process involves injection attack behavior.

6. The method of claim 5, wherein: The first encoder and the second encoder are implemented by a recurrent neural network, or by a network based on a transformer structure.

7. The method according to any one of claims 1 to 6, further comprising, after acquiring the first sensor data and the first image frame corresponding to the latest first moment in the image acquisition process: A first background frame corresponding to the first moment and the first sensor data are stored in a designated storage space, wherein the first background frame is obtained after the foreground object in the first image frame is removed.

8. A device for identifying injection attack behavior, comprising: A first acquisition module is configured to acquire, during an image acquisition process of a mobile terminal, first sensor data and a first image frame corresponding to a first moment in the image acquisition process, wherein the first sensor data corresponds to a sensor in the mobile terminal, and the sensor is used to sense the movement of the mobile terminal: A second acquisition module is configured to acquire second sensor data and second image frames corresponding to N pre-stored second moments, respectively, wherein the N second moments are N moments before the first moment in the image acquisition process; The determination module is configured to determine whether the image acquisition process involves injection attack behavior by using the first sensor data and the first image frame, and the second sensor data and the second image frame corresponding to each of the N second moments.

9. The device of claim 8, wherein: The second image frames corresponding to the N second moments are image frames from which the foreground object is removed; The determination module is specifically configured to remove the foreground object in the first image frame to obtain a first background frame corresponding to the first image frame; Using the first sensor data and the first background frame, as well as the second sensor data and the second image frames corresponding to the N second moments, it is determined whether the image acquisition process involves injection attack behavior.

10. A computing device comprising a memory and a processor, wherein: The memory stores executable codes, and when the processor executes the executable codes, the method according to any one of claims 1 to 7 is implemented.

Images