Device and method

Abstract

The purpose of the present invention is to provide a device and method capable of accurately and easily determining tampering.　In a mobile terminal 100, a control unit 105 functions as: a data acquisition unit that acquires acquisition data of a still image or video captured by a camera; and a sensor value acquisition unit that acquires sensor values indicating external conditions. Furthermore, the control unit 105, by comparing the sensor values with the acquisition data, determines abnormalities in acquisition data of a camera 102. For example, abnormalities in the camera 102 can be determined by acquiring acceleration from the acquisition data and comparing said acceleration with the acceleration that is a sensor value.

Description

Apparatus and method

[0001] The present invention relates to an apparatus and method for determining abnormalities in sensor values.

[0002] Patent Document 1 describes an apparatus for automatically detecting tampering in a network control system, which detects tampering by detecting inconsistencies in configuration data received from a specific network component based on a comparison between the received configuration data and reference configuration data stored in a database.

[0003] JP 2019-133649 A

[0004] However, it is necessary to prepare reference configuration data in advance. Depending on the type of data, it may not be possible to prepare the configuration data in advance, making it impossible to accurately determine whether the device or the data by that device has been tampered with.

[0005] Therefore, an object of the present invention is to provide an apparatus and method that can accurately and easily determine whether a device or data has been tampered with by the device.

[0006] The device disclosed herein includes an acquired data acquisition unit that acquires acquired data, which is an image or video captured by a camera; a sensor value acquisition unit that acquires sensor values ​​that indicate external conditions; and a control unit that determines abnormalities in the acquired data of the camera by comparing the sensor values ​​with the acquired data.

[0007] According to the present invention, it is possible to determine whether there is an abnormality in the camera or its data.

[0008] 1 is a diagram showing an example of use of the mobile terminal 100 according to the first embodiment of the present disclosure. FIG. 1 is a block diagram showing the functional configuration of the mobile terminal 100. FIG. 2 is a specific example of information stored in a storage unit 107. FIG. 3 is a flowchart showing a process for determining whether or not tampering has occurred using a sensor specific to the same terminal. FIG. 4 is a schematic diagram showing a method for acquiring common data from video data acquired by a camera 102. FIG. 5 is a diagram showing the results of FFT analysis of microphone sound. FIG. 6 is a diagram showing a system configuration using the mobile terminal 100 and other mobile terminals 100a, etc. according to a second embodiment. FIG. 7 is a diagram showing the functional configuration of a mobile terminal 100x according to the second embodiment of the present disclosure. FIG. 8 is a diagram showing a specific example of a storage unit 107x. FIG. 9 is a flowchart showing a process for determining whether or not tampering has occurred using the same sensor of a different terminal according to the second embodiment. FIG. 10 is a flowchart showing a process for determining whether or not tampering has occurred using a sensor specific to a different terminal according to the third embodiment. FIG. 11 is a diagram showing an example of the hardware configuration of the mobile terminal 100 according to an embodiment of the present disclosure.

[0009] The present disclosure will be described with reference to the accompanying drawings. Whenever possible, the same parts are designated by the same reference numerals and redundant description will be omitted.

[0010] 1 is a diagram illustrating an example of use of a mobile terminal 100 according to a first embodiment of the present disclosure. As shown in the figure, the mobile terminal 100 according to the present disclosure can operate independently and can determine the possibility of tampering with the camera 102 or the image data of the camera 102 based on image data captured by the camera 102 and sensor values ​​from other sensors in the mobile terminal 100. In the present disclosure, image data includes either still images or videos, unless otherwise specified.

[0011] In addition, the server 200 may acquire image data and sensor values ​​from the camera 102 of the mobile terminal 100 and compare these image data and sensor values ​​to determine whether the camera 102 of the mobile terminal 100 (or its image data) has been tampered with.

[0012] 2 is a block diagram showing the functional configuration of the mobile terminal 100. As shown in the figure, the mobile terminal 100 includes a gyro sensor 101, a camera 102, a microphone 103, an information acquisition unit 104, a control unit 105, an output unit 106, and a storage unit 107. Note that although the gyro sensor 101, the camera 102, and the microphone 103 are given as examples of sensors, it goes without saying that other sensors may also be used. For example, a temperature sensor or a ranging sensor such as LiDAR (Light Detection and Ranging) may be considered.

[0013] The gyro sensor 101 is a sensor that detects the attitude of the mobile terminal 100, and is a sensor that mainly detects acceleration in each direction.

[0014] The camera 102 is a photographing part that photographs still images or moving images, a part that acquires image data, and a sensor that detects images.

[0015] The microphone 103 is a part that collects sound and is a sensor that detects sound.

[0016] The information acquisition unit 104 is a part that acquires sensor values ​​acquired by sensors of other terminals, but is not an essential component in the first embodiment.

[0017] The control unit 105 is the part that determines whether there are any inconsistencies in the sensor values ​​detected by each sensor (gyro sensor 101, camera 102, etc.), and based on that determination, determines whether any of the sensors have been tampered with.

[0018] The output unit 106 is a part that notifies errors and the like based on the determination results in the control unit 105 .

[0019] The storage unit 107 is a part that stores the sensor values ​​acquired by the gyro sensor 101, the camera 102, etc., and image data.

[0020] 3 is a diagram showing a specific example. As shown in the diagram, the storage unit 107 stores the time of acquisition of the sensor value, the position, the sensor value, and image data. The sensor value is, for example, acceleration information acquired by the gyro sensor 101, a temperature acquired by a temperature sensor, etc. In the present disclosure, the sensor value and image data are stored in association with each other, but this is not necessarily limited to this. Either one may be stored.

[0021] The mobile terminal 100 of the present disclosure can determine whether or not the sensors have been tampered with based on inconsistencies or discrepancies between sensor values ​​from sensors within the mobile terminal 100. In the present disclosure, attention is particularly focused on tampering with the camera 102 or its image data.

[0022] 4 is a flowchart showing a process for determining whether or not tampering has occurred using a sensor for each same terminal. In the present disclosure, FIG. 4 is a flowchart showing a process within the mobile terminal 100.

[0023] First, the control unit 105 checks the hardware status of the mobile terminal 100 (S101). Here, the control unit 105 turns off the network connection and performs a process to isolate the mobile terminal 100 from the external environment. This is to prevent malicious remote operation from the outside in real time. The control unit 105 also checks whether the mobile terminal 100 is rooted or jailbroken. In other words, the control unit 105 checks whether an application (program) not authorized by the mobile terminal 100 manufacturer, etc., has been installed or whether administrator privileges have been obtained by a special method. By performing this process, the control unit 105 reduces the possibility that the mobile terminal 100 has not been tampered with or will be tampered with.

[0024] If the control unit 105 determines that rooting or jailbreaking has occurred based on the hardware status, or if it is unable to disconnect from the network, it determines that there is a risk of tampering with the camera 102 or other sensors (such as the gyro sensor 101) (S108). Rooting or jailbreaking involves operating the OS with administrator privileges, which requires checking the authenticity of the OS or app. For this reason, the following APIs are provided. While this embodiment assumes the use of these APIs, other methods may also be used: Android: Determined using the Play Integrity API; iOS: Determined using the DeviceCheck or AppAttest API. The control unit 105 also performs a hacking check of the sensor that acquires content (S102). This process is intended to determine whether sensor hacking by a third party unintentional of the mobile terminal 100's owner has occurred.

[0025] For example, the control unit 105 sends a sample response via a circuit inside the mobile terminal 100 and checks whether the result exceeds or falls below the upper or lower limit of each sensor (such as the gyro sensor 101). This process is performed at the software level, and when a dummy electronic input is sent to the sensor, it checks whether the response value is within the upper or lower limit of the sensor.

[0026] Furthermore, when checking the camera 102, the control unit 105 can confirm the possible range of resolution of the camera by using an API, etc. In this case, if the camera 102 returns a different resolution (e.g., 1920*1920) than the resolution that should originally be available is specified in the catalog specifications of the sensor (e.g., 1440*1080), the control unit 105 can determine that the resolution may have been tampered with.

[0027] Alternatively, various data may be detected and stored as normal data before executing the processing of the present disclosure, and then compared with that data later. That is, the control unit 105 temporarily acquires data from the sensor. Using pre-trained test data, the control unit 105 uses training data, such as the typical values ​​that devices typically return in a given case, and assumes that data outside that range has been tampered with. For example, in a certain region, the temperature sensor acquired by the mobile terminal 100 tends to range from 18 to 25 degrees in October. Therefore, based on the response value of 5 degrees, which is the provisionally acquired data, it can be determined that the sensor has been tampered with. Alternatively, for example, a microphone waveform acquired under normal conditions in a certain environment may be compared with the microphone waveform to compare it with normal conditions.

[0028] Next, the control unit 105 acquires image data captured by the camera 102 from the storage unit 107 (S103), and acquires a sensor value of another sensor (for example, the gyro sensor 101) (S104). Note that although the data is acquired from the storage unit 107 here, it may be acquired from each of the camera 102 and the gyro sensor 101 when operating in real time.

[0029] The control unit 105 acquires common data from the image data acquired by the camera 102 (S105). For example, when a comparison with the sensor value of the gyro sensor 101 is desired, the control unit 105 calculates acceleration data as common data from the image data. In this case, the image data is a moving image, and the acceleration of the mobile terminal 100 can be calculated based on the movement of the moving image.

[0030] The control unit 105 checks for abnormal values ​​based on the acceleration, which is common data (S106). If there is no problem with the abnormal value, the control unit 105 determines that there has been no tampering with the camera 102 or the gyro sensor 101 (S107). For example, the control unit 105 determines that there has been no tampering if the acceleration obtained from the image data of the camera 102 and the acceleration obtained by the gyro sensor 101 match or highly match.

[0031] In this way, it is possible to determine whether the camera 102 or its image data has been tampered with based on the sensor values ​​of other sensors in the mobile terminal 100.

[0032] In steps S104 and S105, the image data captured by the camera 102 can be compared with various common data.

[0033] As one example, the microphone 103 can be used as another sensor. For example, if the content is a video captured by the camera 102, sound may not be necessary for that content. Even in this case, the camera 102 captures the video, the microphone 103 collects the sound, and the control unit 105 records the sound. The control unit 105 then performs frequency analysis of the sound waveform from the microphone 103 using fast Fourier transform (FFT) analysis to check for any waveforms that may cause jamming. At the same time, the control unit 105 checks the video for any suspicious blurs or objects that should not be present and compares them.

[0034] A more detailed description of the process will be given with reference to Figures 5 and 6. Figure 5 is a schematic diagram showing a method for acquiring common data from video data acquired by the camera 102. Figure 5(a) shows one frame in a certain time period of the video data, and shows that the analysis range w is moved and changed within the image F shown in that one frame. Image F is composed of 1280 pixels x 8000 pixels, and the analysis range w is composed of a range of 100 pixels x 100 pixels. The analysis range w is scanned across the entire image F, thereby analyzing the entire image F.

[0035] Although the above analysis process may be performed on all frames that make up the video data, this may result in a huge amount of processing, so it may be better to focus on a specific time period and extract samples, or to process several frames together on an average basis.

[0036] 5B is a diagram showing an analysis of image F with an increased analysis range w. Here, the analysis range w is configured as 300 pixels x 300 pixels.

[0037] In FIGS. 5A and 5B, for example, an FFT analysis is performed on 120 frames of images while changing the position of the analysis range w, and the frequency generated for each position is output as the output.

[0038] The frequency of occurrence of each frequency is then tallied for each analysis range w and its position in the image, and information indicating the temporal transition of the feature values ​​shown in FIG. 5C is generated. If the occurrence frequency is equal to or greater than a predetermined reference value, it is determined that there is an abnormality in that frequency. While FIG. 5C shows one graph, a separate graph is generated for each analysis range w and its position. By changing the size of the analysis range w, it is possible to detect the reflection of objects of different sizes. For example, it may be impossible to determine whether an object is the size of a baseball or a basketball unless the size of the window function is changed.

[0039] 6 is a diagram showing the results of FFT analysis of microphone sound. As shown in the diagram, the frequency of occurrence of a certain frequency over time is shown.

[0040] The control unit 105 can then determine abnormal values ​​by comparing the time transitions shown in FIG. 5(c) with the time transitions shown in FIG. 6. For example, the control unit 105 can determine whether tampering has occurred in the camera 102 or microphone 103 based on the comparison results. For example, the frequency obtained by image analysis represents the appearance rate of objects in the image that are close to the filter size. Furthermore, the frequency obtained by audio analysis represents the presence or absence of jamming at a certain time, i.e., the frequency of a specific frequency. If a time t at which a frequency that appears to be jamming is detected through audio analysis is identified, a search is performed for image frequencies with an abnormal appearance frequency at the certain time t for all image positions (x, y). If such an image frequency with an abnormal appearance frequency is found, it can be determined that tampering by the camera 102 or microphone 103 is possible. A more detailed procedure is as follows: (i) In FIG. 6, n graphs of a specific frequency are examined for trends near frequencies where jamming is likely to occur. (ii) If jamming is suspected (if the threshold is exceeded), check FIG. 5(c) for any frequencies exceeding the threshold at the same time when jamming of the same frequency occurs. (iii) Repeat steps (i) and (ii) above for a number of graphs equal to the number of graphs for the entire image position multiplied by the size of the window function (object). Ideally, checks should be performed for all frequency bands, image position pixels, and window function sizes, but adjusting the sampling rate can reduce the amount of calculation. In step (iii), if the confirmation in step (ii) above is successful in a predetermined number of graphs, it can be determined that tampering is possible. Alternatively, if the frequency occurring during jamming is frequently observed in the data from either the camera 102 or the microphone 103, it can be determined that tampering has occurred in that sensor (either the camera 102 or the microphone 103, or both). Even if both the camera 102 and the microphone 103 exhibit trends that differ from normal trends, it can be determined that tampering has occurred. It is assumed that normal trends are stored in advance.

[0041] In this way, it is possible to determine the possibility of tampering with the camera 102 using the camera 102 and the microphone 103.

[0042] The following method is also possible. If the content is a video captured by the camera 102, data from an illuminance sensor (not shown) is also acquired. The control unit 105 may check whether the results of the luminance analysis of the video match the brightness information obtained by the illuminance sensor. While a video is used here, the same applies to a still image.

[0043] Furthermore, if the content is a video captured by camera 102, the determination may be made by comparing the image captured by camera 102 with the image captured by LiDAR (Light Detection And Ranging).

[0044] For example, the control unit 105 compares the shape of the subject indicated by the sensor value of the mobile terminal 100 with the analysis result (shape of the subject) of the image data acquired by the camera 102 of the mobile terminal 100. If the image data is misaligned, the control unit 105 determines that the camera 102 of the mobile terminal 100 has been tampered with. More specifically, the mobile terminal 100 acquires an image (a so-called point cloud data image) using LiDAR. Then, by analyzing this image, the shape of the subject or the distance to the subject, which is analytical information, is acquired. In the present disclosure, when the shape of the subject is acquired, the mobile terminal 100 also performs image processing to acquire the shape of the subject from the image data. When the distance to the subject is acquired, the mobile terminal 100 also performs image processing to acquire the distance to the subject from the image data.

[0045] In this way, the mobile terminal 100 alone can determine whether sensors including the camera 102 have been tampered with.

[0046] These processes may be performed only once, such as when the mobile terminal 100 is purchased, or may be performed each time data is acquired. In addition, the trigger for re-executing the processes may be specified as, for example, when the location information changes.

[0047] Second Embodiment Next, a method using another mobile terminal will be described. Fig. 7 is a diagram showing a system configuration using the mobile terminal 100 and another mobile terminal 100a in the second embodiment.

[0048] As shown in the figure, mobile terminal 100 is the local terminal, and mobile terminals 100a to 100c are other terminals. Mobile terminal 100 can determine whether camera 102 included in its local terminal has been tampered with based on sensor values ​​detected by the other terminals, mobile terminals 100a to 100c.

[0049] The server 200 is a device that is communicatively connected to the mobile terminal 100, the mobile terminal 100a, etc. via a network. The mobile terminal 100 may acquire the sensor values ​​of the mobile terminal 100a, etc. via the server 200, or may acquire the sensor values ​​directly from the mobile terminal 100a via short-range wireless communication or the like.

[0050] In addition, the server 200 may acquire image data from the camera 102 of the mobile terminal 100 and sensor values ​​of the mobile terminal 100a, etc., and compare these image data and sensor values ​​to determine whether the camera 102 of the mobile terminal 100 (or its image data) has been tampered with.

[0051] In addition, the server 200 may store reference values ​​for each sensor value of the mobile terminal 100 and the like, and determine whether or not tampering has occurred based on these.

[0052] 8 is a diagram illustrating the functional configuration of a mobile terminal 100x according to the second embodiment of the present disclosure. As shown in the figure, the mobile terminal 100x differs from the mobile terminal 100 in that it includes a storage unit 107x. The storage unit 107x stores sensor value information of each of the mobile terminals 100a to 100c. The storage unit 107x is generally the same as the storage unit 107, but the stored contents are different.

[0053] FIG. 9 is a diagram showing a specific example of storage unit 107x. As shown in the figure, the storage unit 107x stores the model, the time when the sensor value was acquired, the position and shooting direction, the sensor value, and image data. The control unit 105 acquires and stores information about other mobile terminals, such as 100a, in this storage unit 107x. Information about the mobile terminal itself may also be stored. The position and shooting direction column indicates that the image was captured at the x, y position, facing the direction indicated by w (westward). As with FIG. 3, the sensor value and image data (still image and video) do not necessarily need to be stored together.

[0054] 10 is a flowchart showing the process of determining whether or not tampering has occurred by the same sensor of another terminal in this second embodiment. This process is based on image data obtained by a terminal other than the mobile terminal 100, such as the mobile terminal 100a, and shows the operation of the mobile terminal 100.

[0055] In advance, each camera of the other terminals, ie, the mobile terminals 100a to 100c, acquires and stores image data and its position information (S201).

[0056] In the mobile terminal 100, the control unit 105 checks the state of the hardware (S101) and performs a hacking check on sensors such as the gyro sensor 101 (S102).

[0057] Thereafter, the control unit 105 acquires image data captured by the camera 102 in accordance with the user's operation (S103).

[0058] Then, the control unit 105 establishes a communication connection with a nearby mobile terminal 100a or the like (S103a), and the mobile terminal 100a transmits image data to the mobile terminal 100. The mobile terminal 100 acquires image data from the mobile terminal 100a or the like (S104a). The control unit 105 transmits location information, shooting direction, and shooting time to the mobile terminal 100a. Based on this information, the mobile terminal 100a transmits image data whose location information, shooting direction, and shooting time match or are roughly the same.

[0059] The control unit 105 checks for abnormalities based on image data of nearby mobile terminals 100 (S106). For example, after acquiring image data at the mobile terminal 100, the control unit 105 checks whether the camera 102 of the mobile terminal 100 has been tampered with based on image data of another terminal, such as the mobile terminal 100a (S106). It is assumed that the mobile terminal 100a has photographed the same object.

[0060] Here, the mobile terminal 100 acquires image data from the mobile terminal 100a via P2P using short-range wireless communication or the like, and can determine whether the image data matches based on the acquired image data. This enables real-time processing. At this time, the image data is associated with the shooting time, location information, and shooting direction. The mobile terminal 100 also acquires image data that was captured within a predetermined time period and whose location information and shooting direction match (or are roughly the same), and uses that image data as the comparison target.

[0061] Then, when the control unit 105 determines in step S106 that there is no abnormality in the image data (S106: OK), it determines that there is no tampering with the camera (S107).

[0062] Furthermore, if the control unit 105 determines that the process is NG in steps S101, S102, and S106, it determines that the camera has been tampered with (S108).

[0063] In this way, processing by the same sensor (camera) can be realized by different terminals.

[0064] Note that the mobile terminals 100 and 100a may upload the acquired image data to the server 200 and compare and verify the data on the server 200. For example, the server 200 may acquire image data including a certain landmark captured by another mobile terminal 100a in advance, and compare this image data with image data including the landmark captured by the mobile terminal 100 to check for tampering. This can be effectively performed by batch processing, and efficient processing can be achieved by using past information stored in the server 200. The confirmation at this time may be performed by analyzing the image on the server 200, and if the degree of match of the feature amounts exceeds 80%, the image may be considered true (no tampering).

[0065] In the third embodiment, in steps S104a and S202 of the second embodiment, the mobile terminal 100a receives sensor values ​​from another sensor instead of image data. In this case, the mobile terminal 100a needs to be in approximately the same position and time as the mobile terminal 100.

[0066] Additionally, as a pre-processing step, various mobile terminals (including the mobile terminal 100 itself) may acquire (learn) sensor values ​​and store them as test cases. For example, the mobile terminal 100 (or the server 200) can compare its own sensor values ​​with those previously acquired (stored) by other mobile terminals 100a to determine whether they match. For example, it can determine whether the sensor values ​​(temperature) from a temperature sensor match. If the match is inconsistent, it can be determined that the sensor in the mobile terminal 100 has been tampered with. Since sensors with the same model number as the mobile terminal 100 are essentially guaranteed to be the same model, this characteristic can be utilized to accurately determine the sensor value. In this case, as with the above, the location and time must be roughly the same.

[0067] The third embodiment will be described in detail below. In the third embodiment, as in the second embodiment, the mobile terminal 100x operates in the system configuration shown in Figures 7 and 8. While the second embodiment compares image data with each other, the third embodiment differs in that it compares image data with a sensor value obtained by a different sensor.

[0068] 11 is a flowchart showing a process for determining whether or not tampering has occurred using a separate terminal sensor. In the present disclosure, FIG. 11 shows a process performed within the mobile terminal 100x.

[0069] In advance, the mobile terminals 100a to 100c, which are other terminals, acquire and store the sensor values ​​from the sensors and their position information (S201a).

[0070] In the mobile terminal 100, the control unit 105 checks the state of the hardware (S101) and performs a hacking check on sensors such as the gyro sensor 101 (S102).

[0071] Thereafter, the control unit 105 acquires image data captured by the camera 102 in accordance with the user's operation (S103).

[0072] Then, the control unit 105 establishes a communication connection with the nearby mobile terminal 100a (S103a), and the mobile terminal 100a transmits the sensor value to the mobile terminal 100 (S202a). The mobile terminal 100 acquires the sensor value from the mobile terminal 100a (S104b).

[0073] The control unit 105 acquires common data from the image data acquired by the camera 102 and the acquired sensor values ​​(S105). In the present disclosure, the common data is as follows, such as the shape of the subject or the distance to the subject.

[0074] For example, the control unit 105 compares the shape of the subject indicated by the sensor value sent from the mobile terminal 100a with the analysis result (shape of the subject) of the image data acquired by the camera 102 of the mobile terminal 100. If the image data is misaligned, the control unit 105 determines that the camera 102 of the mobile terminal 100 has been tampered with. More specifically, the mobile terminal 100a acquires an image (so-called point cloud data image) using LiDAR (Light Detection and Ranging). Then, by analyzing this image, the shape of the subject or the distance to the subject, which is analytical information, is acquired. In the present disclosure, when the shape of the subject is acquired, the mobile terminal 100 also performs image processing to acquire the shape of the subject from the image data. When the distance to the subject is acquired, the mobile terminal 100 also performs image processing to acquire the distance to the subject from the image data.

[0075] The control unit 105 checks for abnormal values ​​based on this common data (such as the shape of the subject) (S106). If the check results are OK, the control unit 105 determines that the camera 102 has not been tampered with (S107). On the other hand, if the control unit 105 determines that the camera 102 has been tampered with in steps S101, S102, and S106, it determines that the camera 102 has been tampered with (S108).

[0076] In this way, the mobile terminal 100 cooperates with the mobile terminals 100a and the like in the vicinity of the mobile terminal 100 to determine whether the camera 102 of the mobile terminal 100 has been tampered with.

[0077] <Application to Server 200 and Other Forms> In the present disclosure shown in the first to third embodiments, the mobile terminal 100 determines whether its camera 102 has been tampered with, but this is not limited to this. The mobile terminals 100 and 100a may upload the above-mentioned check process (process data and determination results) to the server 200 as learning data, and future checks may be performed based on the learning data. For example, the mobile terminals 100 and 100a may upload image data, location information, shooting direction, shooting time, and each sensor value from the camera 102 to the server 200 as learning data. Then, in the future, another mobile terminal may determine whether its camera has been tampered with by determining differences with the uploaded data.

[0078] Furthermore, the process S102 in each of the above embodiments may be executed only once at a predetermined timing, such as when the mobile terminal 100 is purchased, or may be executed each time data is acquired. Furthermore, the trigger for re-executing the process may be specified as, for example, when the location information changes.

[0079] In the above description, whether or not the camera 102 has been tampered with is determined, but this is not limiting, and whether or not other sensors have been tampered with may be determined based on image data from the camera 102 .

[0080] In this disclosure, the mobile terminal 100 is assumed to be a smartphone or the like, but is not limited to this. For example, the above processing can be applied to a device that has both a LiDAR and a camera sensor, such as a car. The processing can also be applied to IoT devices, wearable devices such as smart watches, smart speakers, and the like.

[0081] Furthermore, for example, in a health campaign in which points are awarded based on the amount of vital data recorded in daily life, the following hypothetical tampering actions can be prevented to prevent the user from tampering with sensors and falsifying data even when the user is not actually exercising: Warming the watch that measures vital data to simulate sweating; Applying ultrasonic noise to cause the acceleration sensor to make a false measurement; (Physically) Shaking the watch up, down, left, and right. In contrast, the following methods can be used to detect sensor tampering: Analyzing video constantly captured by a smart speaker (same configuration as the mobile terminal 100) or a surveillance camera instead of the mobile terminal 100, and using object detection processing to detect whether the user is intentionally engaging in deceptive behavior. For example, the control unit 105 can detect specific actions from the video data, such as simply shaking the user's arm or warming the watch at a specific time.

[0082] The control unit 105 then accesses the wearable device worn by the user to acquire vital data for that time period. If the vital data indicates that the user is exercising, the control unit 105 determines that the vital data has been tampered with.

[0083] In addition, a motion sensor or ultrasonic sensor monitors human movement, and if a person's behavior deviates from normal daily activities, the control unit 105 accesses the wearable device and acquires vital data. The control unit 105 uses a heat source sensor inside the home (or while away from home) to check whether the whole body (or arm) is sweating, or whether only the watch is generating heat (or there is a heat source nearby that is higher than body temperature). The control unit 105 determines that the amount of sweat recorded in the vital data at that time has been tampered with.

[0084] In addition, the control unit 105 may check all of the above data against other normal vital data. The above is based on the premise that smart IoT is widespread throughout towns and homes, and devices are connected to each other.

[0085] In addition, with regard to smart glasses, it is possible to prevent sensor tampering when uploading real-world data as content into a VR space by comparing data obtained from a camera capturing the outside world with data obtained from another device, as described above.

[0086] <Operational Effects of the Present Disclosure> Next, operational effects of the mobile terminal 100 according to the first embodiment of the present disclosure and the mobile terminal 100x according to the second and third embodiments will be described.

[0087] The device disclosed herein is an apparatus that includes an image data acquisition unit (corresponding to the control unit 105) that acquires image data captured by the camera 102, a sensor value acquisition unit (corresponding to the control unit 105) that acquires sensor values ​​that indicate external conditions, and the control unit 105 that determines abnormalities in the image data of the camera 102 by comparing the sensor values ​​with video data.

[0088] This device is included in one of the mobile terminals 100, 100x, and the server 200. For example, if the mobile terminal 100 is the device of the present disclosure, the mobile terminal 100 includes a camera 102 that acquires image data and a sensor (e.g., a gyro sensor 101) that detects an external situation and outputs a sensor value.

[0089] Furthermore, if the server 200 is a device disclosed herein, it acquires image data from the mobile terminal 100 and sensor values ​​from the mobile terminal 100a, etc., and uses these to determine whether there is an abnormality in the image data of the camera 102 of the mobile terminal 100.

[0090] The mobile terminal 100 further includes an information acquisition unit 104, which is a communication unit that communicates with another nearby mobile terminal 100a. The control unit 105 acquires the sensor values ​​of the other nearby mobile terminal 100a via the information acquisition unit 104.

[0091] According to these configurations, the mobile terminal 100 or the like can use other sensor values ​​to determine the possibility of tampering with the camera 102 or the possibility of tampering with the image of the camera 102.

[0092] Since it is often not possible to directly compare the image from camera 102 with the sensor value, in the present disclosure, control unit 105 obtains common data indicating the external conditions from the image data and compares the common data with the sensor value to determine whether there is an abnormality in the image data from camera 102.

[0093] For example, if the sensor value indicates acceleration or illuminance, the control unit 105 can obtain the acceleration or illuminance from the image data by performing image analysis on the image data.

[0094] In addition, in the present disclosure, when the sensor value is the sound picked up by the microphone, the control unit 105 acquires the frequency of the image data and the sound by FFT analysis and determines the abnormality of the image data from the frequency. For example, the control unit 105 in the present disclosure determines the abnormality of the image data based on the frequency comparison result.

[0095] As a result, if a frequency that differs from normal trends appears on both devices, or if a frequency specific to jamming appears with a predetermined amplitude (or frequency), it can be determined that an attack (tampering) has occurred on the camera 102. An example of a frequency state specific to jamming is when, under normal circumstances, the amplitude of frequency analysis is 2 in the 100 kHz band (the vertical axis is defined as 10 as the maximum), but when an abnormality occurs, a wave with an amplitude of 6 is seen around 100 kHz. Note that these frequencies and amplitudes are merely examples and are not limited to these.

[0096] In the present disclosure, the sensor value may be reflection data based on the reflected laser light, such as LiDAR. The control unit 105 can determine whether the reflection data matches the image data and determine whether the image data matches, thereby determining whether the camera 102 is malfunctioning.

[0097] In the present disclosure, the control unit 105 acquires the sensor value as well as the acquisition time and acquisition position of the sensor value. The control unit 105 compares the sensor value with image data corresponding to the acquisition time and acquisition position, thereby determining whether an abnormality has occurred in the camera 102.

[0098] <Apparatus and method of the present disclosure> [1] An apparatus comprising: a data acquisition unit that acquires acquired data, which is a still image or video captured by a camera; a sensor value acquisition unit that acquires sensor values ​​that indicate external conditions; and a control unit that determines an abnormality in the acquired data of the camera based on the sensor values ​​and the acquired data.

[0099] [2] The device according to [1], comprising: a camera that acquires acquired data that is a still image or a video; and a sensor that detects an external situation and outputs a sensor value.

[0100] [3] The device according to [1], further comprising a communication unit that communicates with other nearby terminals, wherein the sensor value acquisition unit acquires the sensor value from the other terminals via the communication unit.

[0101] [4] The device according to any one of [1] to [3], wherein the control unit acquires common data indicating the external situation from the acquired data, and compares the common data with the sensor value.

[0102] [5] The device according to [4], wherein the sensor value indicates acceleration or illuminance, and the control unit acquires the acceleration or illuminance from the acquired data.

[0103] [6] The device according to [1], wherein the sensor value is a sound picked up by a microphone, and the control unit acquires a frequency of the acquired data and the sound by FFT analysis, and determines an abnormality in the acquired data from the frequency.

[0104] [7] The device according to [6], wherein the control unit determines whether the acquired data is abnormal based on a result of the comparison of the frequencies.

[0105] [8] The device according to any one of [1] to [3], wherein the sensor value is reflection data based on a laser beam that is reflected when the laser beam is irradiated, and the control unit compares the reflection data with the acquired data.

[0106] [9] The device according to [3], wherein the sensor value acquisition unit acquires the sensor value together with the acquisition time and acquisition position of the sensor value, and the control unit compares the sensor value with acquired data corresponding to the acquisition time and acquisition position.

[0107]

[10] A method comprising: a data acquisition step of acquiring acquired data, which is a still image or a video captured by a camera; a sensor value acquisition step of acquiring a sensor value indicating an external situation; and a control step of determining an abnormality in the acquired data of the camera based on the sensor value and the acquired data.

[0108] <Explanation of Hardware Configuration and Definitions of Terms, etc.> The block diagrams used in the description of the above embodiments show functional blocks. These functional blocks (components) are realized by any combination of at least one of hardware and software. Furthermore, the method of realizing each functional block is not particularly limited. That is, each functional block may be realized using a single device that is physically or logically coupled, or may be realized using two or more physically or logically separated devices that are connected directly or indirectly (e.g., via wire, wirelessly, etc.) and these multiple devices. The functional block may also be realized by combining the single device or multiple devices with software.

[0109] Functions include, but are not limited to, judgment, determination, assessment, calculation, computation, processing, derivation, investigation, search, confirmation, reception, transmission, output, access, resolution, selection, selection, establishment, comparison, assumption, expectation, consideration, broadcasting, notifying, communicating, forwarding, configuring, reconfiguring, allocating, mapping, and assignment. For example, a functional block (component) that performs transmission is called a transmitting unit or transmitter. As mentioned above, there are no particular limitations on how these functions are implemented.

[0110] For example, the mobile terminal 100 according to an embodiment of the present disclosure may function as a computer that performs processing of the sensor abnormality detection method of the present disclosure. Fig. 12 is a diagram illustrating an example of the hardware configuration of the mobile terminal 100 according to an embodiment of the present disclosure. The mobile terminal 100 described above may be physically configured as a computer device including a processor 1001, a memory 1002, a storage 1003, a communication device 1004, an input device 1005, an output device 1006, a bus 1007, etc.

[0111] In the following description, the term "apparatus" can be interpreted as a circuit, a device, a unit, etc. The hardware configuration of the mobile terminal 100 may be configured to include one or more of the apparatuses shown in the drawings, or may be configured to exclude some of the apparatuses.

[0112] Each function of the mobile terminal 100 is realized by loading specific software (programs) onto hardware such as the processor 1001 and memory 1002, causing the processor 1001 to perform calculations, control communication via the communication device 1004, and control at least one of reading and writing data in the memory 1002 and storage 1003.

[0113] The processor 1001 controls the entire computer by running, for example, an operating system. The processor 1001 may be configured by a central processing unit (CPU) including an interface with peripheral devices, a control device, an arithmetic unit, a register, etc. For example, the above-mentioned control unit 105 may be realized by the processor 1001.

[0114] The processor 1001 also reads programs (program codes), software modules, data, etc. from at least one of the storage 1003 and the communication device 1004 into the memory 1002 and executes various processes in accordance with these. The programs used are those that cause a computer to execute at least some of the operations described in the above-described embodiments. For example, the control unit 105 may be implemented by a control program stored in the memory 1002 and running on the processor 1001, and similar implementations may be made for other functional blocks. While the above-described various processes have been described as being executed by a single processor 1001, they may also be executed simultaneously or sequentially by two or more processors 1001. The processor 1001 may be implemented by one or more chips. The programs may also be transmitted from a network via a telecommunications line.

[0115] The memory 1002 is a computer-readable recording medium and may be configured by, for example, at least one of a read-only memory (ROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a random access memory (RAM), etc. The memory 1002 may also be called a register, a cache, a main memory (primary storage device), etc. The memory 1002 can store executable programs (program codes), software modules, etc. for implementing a sensor anomaly detection method according to an embodiment of the present disclosure.

[0116] Storage 1003 is a computer-readable recording medium, and may be composed of at least one of, for example, an optical disk such as a CD-ROM (Compact Disc ROM), a hard disk drive, a flexible disk, a magneto-optical disk (e.g., a compact disk, a digital versatile disk, a Blu-ray (registered trademark) disk), a smart card, a flash memory (e.g., a card, a stick, a key drive), a floppy (registered trademark) disk, a magnetic strip, etc. Storage 1003 may also be referred to as an auxiliary storage device. The above-mentioned storage medium may be, for example, a database, a server, or other appropriate medium including at least one of memory 1002 and storage 1003.

[0117] The communication device 1004 is hardware (transmission / reception device) for communicating between computers via at least one of a wired network and a wireless network, and is also referred to as, for example, a network device, a network controller, a network card, or a communication module. The communication device 1004 may include a high-frequency switch, a duplexer, a filter, a frequency synthesizer, etc. to implement at least one of frequency division duplex (FDD) and time division duplex (TDD). For example, the information acquisition unit 104 and the output unit 106 described above may be implemented by the communication device 1004. The communication device 1004 may be implemented with a transmitter and a receiver that are physically or logically separated from each other.

[0118] The input device 1005 is an input device (e.g., a keyboard, a mouse, a microphone, a switch, a button, a sensor, etc.) that receives input from the outside. The output device 1006 is an output device (e.g., a display, a speaker, an LED lamp, etc.) that outputs to the outside. Note that the input device 1005 and the output device 1006 may be integrated into one device (e.g., a touch panel).

[0119] Furthermore, each device, such as the processor 1001 and the memory 1002, is connected by a bus 1007 for communicating information. The bus 1007 may be configured using a single bus, or may be configured using different buses between each device.

[0120] Furthermore, mobile terminal 100 may be configured to include hardware such as a microprocessor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a programmable logic device (PLD), or a field programmable gate array (FPGA), and some or all of the functional blocks may be realized by the hardware. For example, processor 1001 may be implemented using at least one of these pieces of hardware.

[0121] The notification of information is not limited to the aspects / embodiments described in the present disclosure and may be performed using other methods. For example, the notification of information may be performed by physical layer signaling (e.g., Downlink Control Information (DCI) and Uplink Control Information (UCI)), higher layer signaling (e.g., Radio Resource Control (RRC) signaling, Medium Access Control (MAC) signaling, broadcast information (Master Information Block (MIB) and System Information Block (SIB))), other signals, or a combination thereof. Furthermore, the RRC signaling may be referred to as an RRC message, and may be, for example, an RRC Connection Setup message, an RRC Connection Reconfiguration message, or the like.

[0122] The order of the procedures, sequences, flowcharts, etc. of each aspect / embodiment described in this disclosure may be changed unless it is consistent. For example, the methods described in this disclosure present elements of various steps using an example order, and are not limited to the particular order presented.

[0123] Input and output information may be stored in a specific location (for example, memory) or may be managed using a management table. Input and output information may be overwritten, updated, or added to. Output information may be deleted. Input information may be sent to another device.

[0124] The determination may be made based on a value represented by one bit (0 or 1), a Boolean value (true or false), or a numerical comparison (e.g., comparison with a predetermined value).

[0125] The aspects / embodiments described in this disclosure may be used alone, in combination, or switched depending on the implementation. Notification of predetermined information (e.g., notification that "X is true") is not limited to explicit notification, but may be implicit (e.g., not notifying the predetermined information).

[0126] Although the present disclosure has been described in detail above, it is clear to those skilled in the art that the present disclosure is not limited to the embodiments described herein. The present disclosure can be implemented in modified and altered forms without departing from the spirit and scope of the present disclosure as defined by the claims. Therefore, the description of the present disclosure is intended to be illustrative and does not have any limiting meaning on the present disclosure.

[0127] Software shall be construed broadly to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, threads of execution, procedures, functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise.

[0128] Software, instructions, information, etc. may also be transmitted or received over a transmission medium. For example, if software is transmitted from a website, server, or other remote source using wired technologies (such as coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL)), and / or wireless technologies (such as infrared, microwave), then these wired and / or wireless technologies are included within the definition of transmission media.

[0129] The information, signals, etc. described in this disclosure may be represented using any of a variety of different technologies. For example, data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof.

[0130] Note that terms described in this disclosure and terms necessary for understanding this disclosure may be replaced with terms having the same or similar meanings. For example, at least one of a channel and a symbol may be a signal (signaling). Furthermore, a signal may be a message. Furthermore, a component carrier (CC) may be called a carrier frequency, a cell, a frequency carrier, etc.

[0131] Furthermore, the information, parameters, etc. described in the present disclosure may be expressed using absolute values, may be expressed using relative values ​​from a predetermined value, or may be expressed using other corresponding information. For example, a radio resource may be indicated by an index.

[0132] The names used for the above-described parameters are not intended to be limiting in any way. Furthermore, the mathematical expressions using these parameters may differ from those explicitly disclosed in this disclosure. The various channels (e.g., PUCCH, PDCCH, etc.) and information elements may be identified by any suitable names, and therefore the various names assigned to these various channels and information elements are not intended to be limiting in any way.

[0133] In this disclosure, the terms "Mobile Station (MS)," "user terminal," "User Equipment (UE)," "terminal," and the like may be used interchangeably.

[0134] A mobile station may also be referred to by those skilled in the art as a subscriber station, mobile unit, subscriber unit, wireless unit, remote unit, mobile device, wireless device, wireless communication device, remote device, mobile subscriber station, access terminal, mobile terminal, wireless terminal, remote terminal, handset, user agent, mobile client, client, or some other suitable terminology.

[0135] As used in this disclosure, the terms "determining" and "determining" may encompass a wide variety of actions. "Determining" and "determining" may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, searching, inquiring (e.g., searching in a table, database, or other data structure), ascertaining, and the like. "Determining" and "determining" may also include receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, accessing (e.g., accessing data in memory), and the like. Furthermore, "judgment" and "decision" can include regarding resolving, selecting, choosing, establishing, comparing, etc. as having been "judged" or "decided." In other words, "judgment" and "decision" can include regarding some action as having been "judged" or "decided." Furthermore, "judgment (decision)" can be interpreted as "assuming," "expecting," "considering," etc.

[0136] The terms "connected," "coupled," or any variation thereof, refer to any direct or indirect connection or coupling between two or more elements, and may include the presence of one or more intermediate elements between two elements that are "connected" or "coupled" to each other. The coupling or connection between elements may be physical, logical, or a combination thereof. For example, "connected" may be read as "access." As used in this disclosure, two elements may be considered to be "connected" or "coupled" to each other using one or more wires, cables, and / or printed electrical connections, as well as electromagnetic energy having wavelengths in the radio frequency range, microwave range, and optical (both visible and invisible) range, as some non-limiting and non-exhaustive examples.

[0137] As used in this disclosure, the phrase "based on" does not mean "based only on," unless expressly stated otherwise. In other words, the phrase "based on" means both "based only on" and "based at least on."

[0138] As used in this disclosure, any reference to an element using a designation such as "first," "second," etc. does not generally limit the quantity or order of those elements. These designations may be used in this disclosure as a convenient method of distinguishing between two or more elements. Thus, a reference to a first and a second element does not imply that only two elements may be employed or that the first element must in some way precede the second element.

[0139] When the terms "include," "including," and variations thereof are used in this disclosure, these terms are intended to be inclusive, similar to the term "comprising." Furthermore, when the term "or" is used in this disclosure, it is not intended to be an exclusive or.

[0140] In this disclosure, where articles are added by translation, such as a, an, and the in English, the disclosure may include that the nouns following these articles are in the plural form.

[0141] In the present disclosure, the term "A and B are different" may mean "A and B are different from each other." The term may also mean "A and B are each different from C." Terms such as "separate" and "coupled" may also be interpreted in the same way as "different."

[0142] 100...Mobile terminal, 101...Gyro sensor, 102...Camera, 103...Microphone, 104...Information acquisition unit, 105...Control unit, 106...Output unit, 100a...Mobile terminal, 107...Memory unit, 100x...Mobile terminal, 107x...Memory unit.

Claims

1. A device comprising: a data acquisition unit that acquires acquisition data which is a still image or a moving image captured by a camera; a sensor value acquisition unit that acquires a sensor value indicating an external situation; and a control unit that determines an abnormality in the acquisition data of the camera based on the sensor value and the acquisition data.

2. The device according to claim 1, comprising: a camera that acquires acquisition data which is a still image or a moving image; and a sensor that detects an external situation and outputs a sensor value.

3. The device according to claim 1, further comprising a communication unit that communicates with another terminal in the vicinity, wherein the sensor value acquisition unit acquires a sensor value from the other terminal via the communication unit.

4. The control unit of the device according to claim 1 acquires common data indicating the external situation from the acquisition data, and compares the common data with the sensor value.

5. The sensor value indicates acceleration or illuminance, and the control unit acquires acceleration or illuminance from the acquisition data in the device according to claim 4.

6. The sensor value is voice recorded by a microphone, and the control unit acquires a frequency by performing FFT analysis on the acquisition data and the voice, and determines an abnormality in the acquisition data from the frequency in the device according to claim 1.

7. The control unit of the device according to claim 6 determines an abnormality in the acquisition data based on the comparison result of the frequencies.

8. The sensor value is reflection data based on laser light irradiated and reflected laser light, and the control unit compares the reflection data with the acquisition data in the device according to claim 1.

9. The sensor value acquisition unit acquires, together with the sensor value, the acquisition time and acquisition position of the sensor value, and the control unit compares the acquisition data corresponding to the acquisition time and acquisition position with the sensor value in the device according to claim 3.

10. A method comprising: a data acquisition step of acquiring acquisition data which is a still image or a moving image captured by a camera; a sensor value acquisition step of acquiring a sensor value indicating an external situation; and a control step of determining an abnormality in the acquisition data of the camera based on the sensor value and the acquisition data.

Images