Scene recognition method and device, equipment and storage medium

Abstract

The invention provides a scene recognition method and device, equipment and a storage medium, and the method comprises the steps: extracting data features of sensor data and signal features of a wireless communication signal, the sensor data being data collected by a physical sensor of first equipment; fusing the data features and the signal features to obtain fused features; and recognizing the current scene according to the fused features. By implementing the method, the precision of scene recognition can be improved.

Description

Technical Field

[0001] This application relates to the field of terminal technology, and in particular to a scene recognition method, apparatus, device and storage medium. Background Technology

[0002] In practice, it has been found that users' real-time needs are highly correlated with their activity scenarios. Against this backdrop, as terminal devices develop towards intelligence and user-friendliness, accurately identifying users' activity scenarios has become a key technical means to provide more precise services, and this is also a technical challenge that the industry urgently needs to overcome. Summary of the Invention

[0003] This application provides a scene recognition method, apparatus, device, and storage medium, which can achieve accurate scene recognition.

[0004] The first aspect of this application provides a scene recognition method, which is applicable to a first device and includes: Extract the data features of the sensor data and the signal features of the wireless communication signal, wherein the sensor data is the data collected by the physical sensors of the first device; The data features and the signal features are fused to obtain the fused features; The current scene is identified based on the fused features.

[0005] A second aspect of this application provides a scene recognition device, which is applicable to a first device, and the device includes: A feature extraction unit is used to extract data features of sensor data and signal features of wireless communication signals, wherein the sensor data is data collected by the physical sensors of the first device; A feature fusion unit is used to fuse the data features and the signal features to obtain fused features; A scene recognition unit is used to identify the current scene based on the fused features.

[0006] A third aspect of the embodiments of this application provides an electronic device, Memory containing executable program code; and the processor coupled to the memory; The processor calls the executable program code stored in the memory, and when the executable program code is executed by the processor, the processor implements the method disclosed in the first aspect of the embodiments of this application.

[0007] A fourth aspect of this application provides a computer-readable storage medium having executable program code stored thereon, wherein when the executable program code is executed by a processor, it implements the method disclosed in the first aspect of this application.

[0008] The fifth aspect of this application discloses a computer program product that, when run on a computer, causes the computer to execute the method disclosed in the first aspect of this application.

[0009] The sixth aspect of this application discloses an application publishing platform for publishing computer program products, wherein when the computer program product is run on a computer, the computer executes the method disclosed in the first aspect of this application.

[0010] As can be seen from the above technical solutions, the embodiments of this application have at least the following advantages: Extract data features from sensor data and signal features from wireless communication signals. The sensor data is data collected by the physical sensors of the first device. Fuse the data features and signal features to obtain fused features. Identify the current scene based on the fused features.

[0011] This method extracts data features from sensor data and signal features from wireless communication signals, then fuses these features, and finally identifies the current scene based on the fused features. Since the features used to identify the current scene are multimodal, this improves the accuracy of scene recognition. Attached Figure Description

[0012] Figure 1 This is a flowchart illustrating a scene recognition method disclosed in an embodiment of this application; Figure 2 This is another flowchart illustrating the scene recognition method disclosed in the embodiments of this application; Figure 3 This is another flowchart illustrating the scene recognition method disclosed in the embodiments of this application; Figure 4A This is an interface illustration of an output prompt message disclosed in an embodiment of this application; Figure 4B This is an illustration of a push settings interface disclosed in an embodiment of this application; Figure 4C This is another illustration of the push settings interface disclosed in the embodiments of this application; Figure 4D This is yet another illustration of the push settings interface disclosed in the embodiments of this application; Figure 5 This is another flowchart illustrating the scene recognition method disclosed in the embodiments of this application; Figure 6 This is another flowchart illustrating the scene recognition method disclosed in the embodiments of this application; Figure 7 This is a structural illustration of a scene recognition device disclosed in an embodiment of this application; Figure 8 This is a structural illustration of an electronic device disclosed in an embodiment of this application. Detailed Implementation

[0013] This application provides a scene recognition method, apparatus, device, and storage medium, which can achieve accurate scene recognition.

[0014] To enable those skilled in the art to better understand the present application, the technical solutions of the embodiments of the present application will be described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. All embodiments based on the present application should fall within the scope of protection of the present application.

[0015] It should be noted that, in this application, the words "exemplarily" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplarily" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of words such as "exemplarily" or "for example" is intended to present the relevant concepts in a specific manner.

[0016] "At least one" means one or more, while "more" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, and c can mean: a, or b, or c, or a and b, or a and c, or b and c, or a, b, and c, where a, b, and c can be single or multiple.

[0017] In related technologies, the identification of user activity scenarios is usually achieved based on GPS positioning technology. The specific implementation process typically involves first calculating the geographical location based on the received satellite signals, and then inferring the user's current activity scenario based on the calculated geographical location. However, in typical environments such as indoors and densely populated urban areas, the satellite signals received by the device are often very weak, which usually leads to poor positioning accuracy of the device, thus affecting the accuracy of scene recognition.

[0018] To address this issue, this application discloses a scene recognition method, apparatus, device, and storage medium, which helps improve the accuracy of scene recognition. In this scene recognition method, a first device extracts data features from sensor data and signal features from wireless communication signals, then fuses the data features and signal features, and finally identifies the current scene based on the fused features. The sensor data is collected by the physical sensors of the first device.

[0019] In this application, the fusion feature used to identify the current scene is a multimodal feature composed of data features from sensor data and signal features from wireless communication signals. Furthermore, compared to satellite signals, both sensor data and wireless communication signals are relatively reliable. Therefore, the data features extracted from sensor data and the signal features extracted from wireless communication signals can realistically reflect the current scene. It is evident that the fusion feature used to identify the current scene is not only multimodal but also realistically reflects the current scene, which is beneficial for improving the accuracy of scene identification.

[0020] The physical sensors of the first device may include, but are not limited to, at least one of the following: an inertial measurement unit, a barometer, a microphone, etc. The wireless communication signal is a signal collected by the communication module of the first device. The communication module of the first device may be a cellular network module or a near-field communication module; this application embodiment does not limit this. The near-field communication module may include, but is not limited to, a Bluetooth module, a Wireless Fidelity (Wi-Fi) module, and a Near Field Communication (NFC) module, etc.

[0021] The electronic devices (first devices) involved in the embodiments of this application may include general handheld screen electronic devices, such as mobile phones, smartphones, portable terminals, terminals, personal digital assistants (PDAs), portable multimedia players (PMPs), laptops, notebooks, wireless broadband (Wibro) terminals, tablet computers (PCs), smart PCs, point of sale (POS) terminals, etc.

[0022] Electronic devices can also include wearable devices. Wearable devices are portable electronic devices that can be worn directly on the user's body or integrated into the user's clothing or accessories. Wearable devices are not just hardware devices; they can also achieve powerful intelligent functions through software support and data interaction and cloud interaction, such as computing, positioning, and alarm functions. They can also connect to mobile phones and various terminals. Wearable devices can include, but are not limited to, wrist-supported devices (such as watches, wristbands, etc.), foot-supported devices (such as shoes, socks, or other leg-wearing products), head-supported devices (such as glasses, helmets, headbands, etc.), as well as smart clothing, backpacks, canes, accessories, and other non-mainstream product forms.

[0023] The scene recognition method disclosed in this application will be described in detail below with reference to specific implementation methods.

[0024] Please see Figure 1 , Figure 1 This is a flowchart illustrating a scene recognition method disclosed in an embodiment of this application. Figure 1 The scene recognition method shown may include the following steps: 101. The first device extracts the data characteristics of the sensor data and the signal characteristics of the wireless communication signal.

[0025] The aforementioned sensor data may be data collected by the physical sensors of the first device, which may include one or more physical sensors. It is understood that the aforementioned sensor data may be data collected by one physical sensor or data collected by multiple physical sensors; this application embodiment does not impose such limitations.

[0026] In embodiments of this application, the physical sensor of the first device may include, but is not limited to, at least one of the following: an inertial measurement unit (IMU), a barometer, and a microphone. For example, when the physical sensor is an IMU, the sensor data may be acceleration; further example, when the physical sensor is an IMU and a barometer, the sensor data may include acceleration and air pressure; still example, when the physical sensor is an IMU and a microphone, the sensor data may include acceleration and ambient audio; and still example, when the physical sensor includes an IMU, a barometer, and a microphone, the sensor data may include acceleration, air pressure, and ambient audio.

[0027] Among these features, acceleration data can be variance, which can be used to detect the user's motion state (e.g., stationary or moving). Barometric pressure data can be the rate of change of barometric pressure, which can be used to detect floor transitions. Ambient audio data can be spectral features, which can be used to detect ambient sounds.

[0028] The wireless communication signal can be a signal detected by the communication module of the first device. The wireless communication signal can include cellular signals and / or short-range communication signals (such as Bluetooth signals, Wi-Fi signals, etc.). The communication module of the first device can include a cellular network module and / or a short-range communication module. The short-range communication module can include, but is not limited to, Bluetooth modules, Wi-Fi modules, and NFC modules.

[0029] The signal characteristics of a wireless communication signal may include a signal identifier and / or a signal change trend. The signal identifier is used to uniquely identify a signal and may consist of at least one of the following: numbers, letters, and special characters.

[0030] Understandably, the signal identifier of a wireless communication signal can be used to assist in scene identification. The identifier of a Bluetooth signal can be used to identify the category of the device broadcasting the Bluetooth signal, and based on this category, the first device can identify the current scene. For example, when the first device identifies the device broadcasting the Bluetooth signal as a home appliance (such as an air conditioner, refrigerator, lamp, speaker, robot vacuum cleaner, etc.) by recognizing the signal identifier of the currently detected Bluetooth signal, the first device determines the current scene to be at home. As another example, when the first device identifies the device broadcasting the Bluetooth signal as a vehicle infotainment system by recognizing the signal identifier of the currently detected Bluetooth signal, the first device determines the current scene to be a driving scenario.

[0031] The identifier of a Wi-Fi signal can be used to identify the usage scenario of that Wi-Fi signal. For example, subways have dedicated Wi-Fi, high-speed trains have dedicated Wi-Fi, airplanes have dedicated Wi-Fi, and in-vehicle systems have dedicated Wi-Fi. For instance, when a first device determines that the Wi-Fi signal is coming from a subway by recognizing its identifier, the first device determines that the current scenario is subway commuting. As another example, when the first device determines that the Wi-Fi signal is coming from an in-vehicle system by recognizing its identifier, the first device determines that the current scenario is driving.

[0032] Understandably, the signal variation trend of wireless communication signals can also be used to assist in scene identification. For example, the signal variation trend of cellular network signals in a subway is a pulse-like change of "weak -> strong -> weak". When the first device detects that the signal variation trend of the cellular network signal is a pulse-like change of "weak -> strong -> weak", the first device determines that the current scene is subway commuting.

[0033] 102. The first device fuses data features and signal features to obtain fused features.

[0034] The first device fuses data features and signal features to obtain fused features, which may include: the first device splicing data features and signal features to obtain spliced ​​features, and performing a dimensionality reduction operation on the spliced ​​features to obtain fused features.

[0035] In some embodiments, the first device can perform dimensionality reduction on the stitched features using an edge artificial intelligence (AI) model to obtain fused features. The edge AI model can be a lightweight neural network or a decision tree; this application embodiment does not limit the specific model.

[0036] The dimension of the fused features can be less than or equal to 128, or even less than a lower dimension; this application does not limit this.

[0037] Understandably, when fusing data features and signal features, the first device can first perform a simple concatenation of the data features and signal features to obtain a high-dimensional and noisy feature vector. Then, by reducing the dimensionality of this feature vector, the most discriminative features are extracted to form a low-dimensional and non-redundant feature vector. This helps to reduce the amount of data processing and thus improve the efficiency of scene recognition.

[0038] In one possible implementation, after the first device performs a dimensionality reduction operation on the splicing features, it can further process the dimensionality-reduced splicing features based on locality-sensitive hashing (LSH) technology to obtain the fused features.

[0039] It should be noted that locality-sensitive hashing (LSH) can be used to deduplicate features, which helps improve the efficiency of feature comparison.

[0040] 103. The first device identifies the current scene based on the fusion features.

[0041] In this embodiment, the first device identifies the current scene based on the fused features by using a preset scene recognition model. The scene recognition model can be deployed on the first device or in the cloud; this embodiment does not impose any limitations.

[0042] Understandably, when the scene recognition model is deployed on the first device, the first device directly identifies the current scene based on the fused features using the scene recognition model. When the scene recognition model is deployed in the cloud, the first device sends the fused features to the cloud, and the cloud obtains the current scene based on the fused features using the scene recognition model, and then sends information indicating the current scene back to the first device.

[0043] When the scene recognition model is deployed on the first device, its training can be performed on the first device or in the cloud; this embodiment does not impose any limitation. When the scene recognition model is trained on the first device, it can be trained based on multiple historical scene information from users of the first device. When the scene recognition model is trained in the cloud, it can be trained based on historical scene information from multiple users. Each piece of historical scene information may include a scene type and corresponding fusion features.

[0044] When a scene recognition model is deployed in the cloud, it can be trained based on historical scene information from multiple users.

[0045] It should be noted that when the scene recognition model is deployed on the first device, and the training of the scene recognition model is also performed on the first device, the user's historical behavior information on the first device is stored and processed only on the first device. That is, the entire scene recognition process on the first device is completed only on the first device, without the participation of other devices, thus achieving privacy-preserving scene recognition. When the scene recognition model is deployed in the cloud, the training of the model and the feature-based scene recognition operations are both performed in the cloud, which helps to alleviate the data storage and computing pressure on the first device.

[0046] In the embodiments of this application, the first device may identify the scene continuously or based on an event-driven mechanism; this embodiment of the application does not impose any limitations.

[0047] When the first device recognizes the scene based on an event-driven mechanism, the first device can trigger the execution of step 101 when it detects a change in the scene, which helps to reduce the computational overhead of the first device.

[0048] In some possible implementations, the first device can detect whether the scene has changed using low-power sensors such as pyroelectric infrared sensors, millimeter-wave radar sensors, and ultrasonic sensors.

[0049] Understandably, the first device uses low-power sensors to monitor the scene most of the time, and only performs complex scene recognition when the scene changes significantly, which helps to reduce the power consumption of the first device.

[0050] In some embodiments, the first device may also output notification information indicating the current scene.

[0051] The notification information can be output in at least one of the following ways, including but not limited to: voice, text, and animation.

[0052] In some embodiments, after outputting a notification message, the first device may also collect user feedback information to further optimize the scene recognition model based on the feedback information.

[0053] By implementing Figure 1 The scene recognition method shown involves a first device extracting data features from sensor data and signal features from wireless communication signals, then fusing the data features and signal features, and finally recognizing the current scene based on the fused features. Since the data features used to recognize the current scene are multimodal, this method helps improve the accuracy of scene recognition.

[0054] In this embodiment, the first device can also quickly identify the current scene based on fusion features using a preset scene database. This scene identification method can be combined with the following... Figure 2 Let me introduce it.

[0055] Please see Figure 2 , Figure 2 This is another flowchart illustrating the scene recognition method disclosed in the embodiments of this application. For example... Figure 2 The scene recognition method shown may include the following steps: 201. The first device extracts the data characteristics of the sensor data and the signal characteristics of the wireless communication signal.

[0056] It should be noted that the descriptions of sensor data, sensor data characteristics, wireless communication signals, and wireless communication signal characteristics can be found in the description of step 101 in the above embodiments, and will not be repeated here.

[0057] In this embodiment, to reduce the power consumption of the first device, the first device can determine which physical sensors to activate for data collection based on the current power level. Therefore, before step 201, the first device can also obtain its current power level, and if the current power level is less than a first power threshold, collect sensor data through a target physical sensor whose power consumption per unit time is less than or equal to a second power threshold.

[0058] The first power threshold and the second power threshold are both preset power values. They can be set by the first device at the factory or by the first device during the user's use. This application embodiment does not limit this.

[0059] For example, if the microphone consumes less power than the inertial measurement unit (IMU), the first device can activate the microphone to collect ambient audio when the battery is low. Alternatively, the first device can activate the IMU to collect acceleration when the battery is high, or it can activate both the microphone and the IMU to collect acceleration.

[0060] It is understandable that when the first device has low battery power, it uses low-power physical sensors to collect sensor data, which helps to reduce the power consumption of the first device and thus helps to extend the battery life of the first device.

[0061] 202. The first device fuses data features and signal features to obtain fused features.

[0062] It should be noted that for a detailed explanation of step 202, please refer to the description of step 102 in the above embodiments, which will not be repeated here.

[0063] 203. The first device searches for target reference features that match the fusion features from the scene database.

[0064] The scene database may include at least one reference feature and at least one scene, with one scene corresponding to one or more reference features. This scene database can be obtained by recording multiple historical scene information of the user of the first device. Each historical scene information includes the scene type and the fusion feature corresponding to each scene. For example, the scene database may be as shown in Table 1 below.

[0065]

[0066] Table 1 In this embodiment, matching the fused feature with the target reference feature means that the similarity between the fused feature and the target reference feature is greater than a similarity threshold. The similarity threshold is preset and can be 75%, 80%, or 85%, etc., and is not limited in this embodiment.

[0067] 204. The first device determines the scene corresponding to the target reference feature as the current scene.

[0068] Understandably, the first device can compare the fused features with reference features in the scene database to obtain a target reference feature that matches the fused feature, and take the scene corresponding to the target reference feature as the current scene.

[0069] For example, if the first device finds a reference feature 2 that matches the fusion feature in the scene database shown in Table 1, then the first device determines that the current scene is the scene corresponding to the reference feature 2, that is, the current scene is an office.

[0070] For example, if the first device finds a reference feature 1 that matches the fused feature in the scene database shown in Table 1, then the first device determines that the current scene is the scene corresponding to the reference feature 1, that is, the current scene is subway commuting.

[0071] In some embodiments, the scene database may further include contextual information associated with each scene, which may include time information and / or location information corresponding to each scene. For example, the scene database can be found in Table 2 below.

[0072]

[0073] Table 2 Based on this, the first device determines the scene corresponding to the target reference feature as the current scene, which may include: the first device matching the target context information associated with the scene corresponding to the target reference feature with the current context information, and determining the scene corresponding to the target reference feature as the current scene when the target context information matches the current context information.

[0074] Understandably, when performing scene recognition, the first device not only considers the fused features obtained so far, but also the current contextual information, which is beneficial to further improve the accuracy of scene recognition.

[0075] For example, the first device finds reference feature 2 that matches the fusion feature from the scene database shown in Table 2. The first device then determines whether the current time is between 9:00 and 18:00. If it is, the first device determines that the current scene is the scene corresponding to reference feature 2, that is, the current scene is an office.

[0076] For example, the first device finds reference feature 1 that matches the fusion feature from the scene database shown in Table 1. The first device then determines whether the current time is between 8:00 and 8:30. If it is, the first device determines that the current scene is the scene corresponding to reference feature 1, that is, the current scene is subway commuting.

[0077] In one possible implementation, the first device can also identify the current scene based on fused features and current context information using a scene recognition model.

[0078] When the scene recognition model is deployed on the first device, the first device can directly identify the current scene based on the fused features and current context information through the scene recognition model.

[0079] When the scene recognition model is deployed in the cloud, the first device can send the fused features and current context information to the cloud. The cloud uses the scene recognition model to obtain the current scene based on the fused features and current context information, and sends the information used to indicate the current scene to the first device.

[0080] In one possible implementation, the data in the scenario database can be encrypted to protect sensitive data.

[0081] In another possible implementation, the scenario database can be a relational database implemented using lightweight SQLite.

[0082] In this embodiment of the application, after identifying the current scene, the first device can also push content based on the identified current scene to achieve personalized content push. The following describes how... Figure 3 A detailed introduction will be provided.

[0083] Please see Figure 3 , Figure 3 This is another flowchart illustrating the scene recognition method disclosed in the embodiments of this application. For example... Figure 3 The scene recognition method shown may include the following steps: 301. The first device extracts the data characteristics of the sensor data and the signal characteristics of the wireless communication signal.

[0084] 302. The first device fuses data features and signal features to obtain fused features.

[0085] 303. The first device identifies the current scene based on the fusion features.

[0086] It should be noted that for detailed information on steps 301-303, please refer to the above. Figure 1-2 The descriptions in the illustrated embodiments will not be repeated here.

[0087] 304. The first device acquires the user's historical behavior in the current scenario.

[0088] In this context, a user's historical behavior in the current scenario can refer to the user's previous application usage behavior in the current scenario, such as playing games, browsing short videos, listening to music, reading news, etc.

[0089] 305. The first device pushes content related to historical behavior.

[0090] Content related to historical behavior may include application resources of the application related to the historical behavior and / or prompt messages used to trigger the opening of the application related to the historical behavior. The push notification method for application resources related to historical behavior can be silent push, and the push notification method for prompt messages used to trigger the opening of the application related to historical behavior can be notification bar push.

[0091] For example, when the historical behavior is playing a game, the content related to the historical behavior may include game resources of the game application and / or a prompt message for launching the game application. The first device can push game-related content in the following ways: it can silently push (preload) game resources of the game application in the background, it can output a prompt message for launching the game application in the notification bar, or it can preload game resources in the background and output a prompt message for launching the game application in the notification bar. The interface for outputting the prompt message for launching the game application can be found below. Figure 4A ,like Figure 4A The interface shown includes a prompt message 410 for triggering the start of the game.

[0092] When a user's historical behavior in the current scenario is multiple, the first device pushes content related to the historical behavior, which may include: The first device acquires the frequency of each historical behavior among multiple historical behaviors; The first device determines the priority of pushing content related to each historical behavior based on the frequency of each historical behavior. The first device pushes at least one type of content related to a historical behavior according to the push priority of each type of historical behavior.

[0093] It's important to note that the more frequent a historical behavior is, the higher the priority of related content push notifications; conversely, the less frequent a historical behavior is, the lower the priority of related content push notifications. For example, if a user's historical behaviors in the current scenario include playing games, browsing short videos, and listening to music, and they played games 15 times, browsed short videos 5 times, and listened to music 3 times, then content related to playing games would have the highest priority, followed by content related to browsing short videos, and content related to listening to music would have the lowest priority.

[0094] The content related to at least one of the above historical behaviors can be content related to each of the multiple historical behaviors, or content related to some of the multiple historical behaviors (content with the highest push priority or the one with the highest push priority).

[0095] For example, when the content related to historical behavior is application resources of the application related to the historical behavior, and the historical behavior includes playing games, browsing short videos, and listening to music, and the push priority of game resources is the highest, the push priority of short video resources is the second highest, and the push priority of audio resources is the lowest, the first device can first preload game resources in the background, then preload short video resources in the background, and finally preload audio resources in the background; or, the first device can also preload game resources only in the background; or, the first device can also preload game resources in the background first, and then preload short video resources in the background.

[0096] In this embodiment of the application, the first device first determines the push priority of content related to each historical behavior based on the frequency of each historical behavior, and then pushes at least one piece of content related to each historical behavior according to the push priority of the content related to each historical behavior, which can make the content push more accurate.

[0097] In one possible implementation, when a user's historical behavior in the current scenario is multiple, the first device can also push content related to each historical behavior simultaneously.

[0098] For example, when the content related to historical behavior is application resources of the application related to the historical behavior, and the historical behavior includes playing games, browsing short videos, and listening to music, the first device can simultaneously preload game resources, short video resources, and audio resources in the background.

[0099] In one possible implementation, the first device can silently push content related to historical behavior when its status information meets the target conditions. The status information may include, but is not limited to, at least one of the following: battery level, network speed, CPU utilization, foreground applications running, and background applications running.

[0100] It should be noted that when the status information of the first device meets the target conditions, it indicates that the current battery level and / or performance of the first device are good. In this case, preloading content related to historical behavior by the first device usually will not affect the use of currently running applications. Conversely, when the status information of the first device does not meet the target conditions, it indicates that the battery level and / or performance of the first device are poor. In this case, preloading content related to historical behavior by the first device often affects the use of currently running applications. Understandably, preloading content related to historical behavior when the first device's status information meets the target conditions can avoid affecting the use of currently running applications.

[0101] It should be noted that the push notification method for content related to historical behavior can be preset by the user. Taking game applications as an example, users can set the game application to push notifications to the notification bar when based on the scenario, such as... Figure 4B As shown; users can also configure game applications to send silent push notifications when using scenario-based push notifications, such as... Figure 4C As shown; users can also configure game applications to either silently push notifications or send notifications through the notification bar when based on scene recognition, such as... Figure 4D As shown.

[0102] By implementing Figure 3The method described involves a first device extracting data features from sensor data and signal features from wireless communication signals, then fusing the data and signal features, and finally identifying the current scene based on the fused features. Since the data features used to identify the current scene are multimodal, this improves the accuracy of scene recognition. Furthermore, after identifying the current scene, the first device can also push content related to the user's historical behavior in that scene to achieve personalized content delivery.

[0103] Based on the above description, the embodiments of this application can provide, as follows: Figure 5 The scene recognition method shown is as follows: Figure 5 The scene recognition method shown may include the following steps: 501. The first device extracts the data characteristics of the sensor data and the signal characteristics of the wireless communication signal.

[0104] 502. The first device fuses data features and signal features to obtain fused features.

[0105] 503. The first device searches for target reference features that match the fusion features in the scene database.

[0106] 504. The first device matches the target context information associated with the scene corresponding to the target reference feature with the current context information.

[0107] 505. When the target context information matches the current context information, the first device determines the scene corresponding to the target reference feature as the current scene.

[0108] 506. The first device acquires the user's historical behavior in the current scenario.

[0109] 507. The first device pushes content related to historical behavior.

[0110] It should be noted that the descriptions of steps 501-507 can be found in the relevant descriptions in the above embodiments, and will not be repeated here.

[0111] In one possible implementation, the first device can push content related to historical behavior via notification bar push when the first device is in a screen-on state.

[0112] In this embodiment of the application, after identifying the current scene, the first device can also implement an anomaly alarm based on the current scene. The following is in conjunction with... Figure 6 An example is provided.

[0113] Please see Figure 6 , Figure 6 This is another flowchart illustrating the scene recognition method disclosed in the embodiments of this application. For example... Figure 6The scene recognition method shown may include the following steps: 601. The first device extracts the data characteristics of the sensor data and the signal characteristics of the wireless communication signal.

[0114] 602. The first device fuses data features and signal features to obtain fused features.

[0115] 603. The first device identifies the current scene based on the fusion features.

[0116] It should be noted that for a detailed description of steps 601-603, please refer to the relevant descriptions in the above embodiments, which will not be repeated here.

[0117] 604. When the current scene differs from the target scene, the first device sends an alarm message to the second device.

[0118] The user of the first device is the ward, and the user of the second device is the guardian. For example, the user of the first device is a child, and the user of the second device is a parent; or, the user of the first device is an elderly person, and the user of the second device is a child.

[0119] The target scenario refers to the activity scenarios that the user of the first device frequently engages in at the current time.

[0120] Understandably, the first device can record at least one frequently occurring activity scenario for a user, as well as the activity time for each scenario. Based on this, after identifying the current scenario, the first device can also obtain the target scenario, i.e., the user's frequently occurring activity scenario at the current time, according to the current time.

[0121] If the current scenario differs from the target scenario, it indicates that the user of the first device has deviated from their behavioral rules. In this case, the first device sends an alert to the second device, which helps the user of the second device to understand the situation in a timely manner.

[0122] For example, the user of the first device is a child, and the user of the second device is a parent. The first device identifies the current scene as a subway and the current time as 9 o'clock. According to the records, the first device knows that the user's activity scene at 9 o'clock is school. The first device then determines that the child has deviated from his behavior rules and sends an alarm message to the second device so that the parent can know in time that the child is not at school but on the subway, which helps to reduce the probability of the child being in danger.

[0123] For example, if the user of the first device is an elderly person and the user of the second device is their children, and the first device identifies the current scene as a secluded park and the current time as 10 o'clock, and the first device knows from the records that the user's activity scene at 10 o'clock was a vegetable market, then the first device determines that the elderly person has deviated from their behavioral rules and sends an alarm message to the second device so that the children can know about the elderly person's situation in time, which helps to reduce the probability of the elderly person being in danger.

[0124] For example, the user of the first device is an elderly person, and the user of the second device is their children. The first device identifies the current scene as a riverside and the current time as 9 PM. Based on the records, the first device knows that the user's activity scene at 9 PM is at home. The first device then determines that the elderly person has deviated from their behavioral rules and sends an alarm message to the second device so that the children can know in time that the elderly person is not at home.

[0125] By implementing Figure 6 The method described involves a first device extracting data features from sensor data and signal features from wireless communication signals, then fusing the data and signal features, and finally identifying the current scene based on the fused features. Since the data features used to identify the current scene are multimodal, this improves the accuracy of scene recognition. Furthermore, after identifying the current scene, the first device can compare it with the user's frequently occurring activity scenarios at the current time. If they are different, an alarm message is sent to a second device so that the user of the second device is promptly aware that the user of the first device has deviated from their behavioral rules.

[0126] Please see Figure 7 , Figure 7 This is a structural illustration of a scene recognition device disclosed in an embodiment of this application. For example... Figure 7 The scene recognition device shown may include a feature extraction unit 701, a feature fusion unit 702, and a scene recognition unit 703; wherein: The feature extraction unit 701 is used to extract data features of sensor data and signal features of wireless communication signals. The sensor data is data collected by the physical sensors of the first device. The feature fusion unit 702 is used to fuse data features and signal features to obtain fused features; The scene recognition unit 703 is used to identify the current scene based on the fused features.

[0127] In some embodiments, the scene recognition unit 703 may specifically identify the current scene based on the fusion features by means of: the scene recognition unit 703 searching for a target reference feature that matches the fusion features from the scene database; and determining the scene corresponding to the target reference feature as the current scene; wherein the scene database includes at least one reference feature and at least one scene, and one scene corresponds to one or more reference features.

[0128] In some embodiments, the scene database further includes context information associated with each scene, and the context information associated with each scene includes time information and / or location information corresponding to each scene; the scene recognition unit 703 is used to determine the scene corresponding to the target reference feature as the current scene in a specific manner, which may include: the scene recognition unit 703 is used to match the target context information associated with the scene corresponding to the target reference feature with the current context information; and, when the target context information matches the current context information, the scene corresponding to the target reference feature is determined as the current scene.

[0129] In some embodiments, the scene recognition unit 703 may be used to recognize the current scene based on the fusion features in a specific manner. The scene recognition unit 703 is used to recognize the current scene based on the fusion features and the current context information through the scene recognition model. The current context information includes the current time information and / or location information.

[0130] In some embodiments, the feature extraction unit 701 is further configured to obtain the current power level of the first device before extracting the data features of the sensor data and the signal features of the wireless communication signal; and, if the current power level is less than a first power level threshold, to collect sensor data through the target physical sensor, wherein the power consumption of the target physical sensor per unit time is less than or equal to a second power level threshold.

[0131] In some embodiments, Figure 7 The scene recognition device shown may also include a content push unit ( Figure 7 (Not shown), a content push unit is used for the scene recognition unit 703 to identify the current scene based on the fusion features, obtain the user's historical behavior in the current scene, and push content related to the historical behavior.

[0132] In some embodiments, the method by which the content push unit pushes content related to historical behavior may specifically include: the content push unit is configured to obtain the frequency of each historical behavior among a variety of historical behaviors; and, based on the frequency of each historical behavior, determine the push priority of content related to each historical behavior; and, according to the push priority of content related to each historical behavior, push at least one type of content related to historical behavior.

[0133] In some embodiments, the push methods for content related to historical behavior include notification push and / or silent push.

[0134] In some embodiments, the content related to historical behavior includes content related to the first historical behavior, and the push method for the content related to the first historical behavior is silent push. The specific method for the content push unit to push the content related to historical behavior may include: the content push unit is used to silently push the content related to the first historical behavior when the status information of the first device meets the target conditions. The status information includes at least one of the following: battery level, network speed, CPU utilization, foreground running applications, and background running applications.

[0135] In some embodiments, Figure 7 The scene recognition device shown may also include an alarm unit ( Figure 7 (Not shown), an alarm unit is used by the scene recognition unit 703 to identify the current scene based on the fusion features, and to send alarm information to the second device when the current scene is different from the target scene.

[0136] Please see Figure 8 , Figure 8 This is a structural illustration of an electronic device disclosed in an embodiment of this application. For example... Figure 8 The electronic device shown may include components such as processor 801, memory 802, display unit 803, input unit 804, sensor 805, and audio circuit 806.

[0137] The processor 801 is the control center of the electronic device. It connects various parts of the electronic device via various interfaces and lines. By running or executing software programs and / or modules stored in the memory 802, and by calling data stored in the memory 802, it performs various functions and processes data, thereby providing overall monitoring of the electronic device. Optionally, the processor 801 may include one or more processing units; optionally, the processor 801 may integrate an application processor, which mainly handles operating devices, user interfaces, and application programs. Of course, it may also include other processors, which are not listed here.

[0138] The memory 802 can be used to store software programs and modules. The processor 801 executes various functional applications and data processing of the electronic device by running the software programs and modules stored in the memory 802. The memory 802 may mainly include a program storage area and a data storage area. The program storage area may store the operating device and application programs required for at least one function (such as sound playback function, image playback function, etc.); the data storage area may store data created according to the use of the electronic device (such as audio data, telephone book, etc.). In addition, the memory 802 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0139] The display unit 803 can be used to display information input by the user or information provided to the user, as well as various menus of the electronic device. The display unit 803 may include a display panel, optionally configured as a Liquid Crystal Display (LCD), Organic Light-Emitting Diode (OLED), or similar display panel. Furthermore, a touch panel may cover the display panel. When the touch panel detects a touch operation on or near it, it transmits the information to the processor 801 to determine the type of touch event. Subsequently, the processor 801 provides corresponding visual output on the display panel based on the type of touch event. The touch panel and the display panel are not located on the same surface. Figure 8 As shown above, the touch panel and display panel can be used as two separate components to realize the input and output functions of the electronic device, or they can be integrated to realize the input and output functions of the electronic device.

[0140] Input unit 804 can be used to receive input digital or character information and generate key signal inputs related to user settings and function control of the electronic device. Specifically, input unit 804 may include a touch panel and other input devices. A touch panel, also known as a touchscreen, can collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel) and drive corresponding connection devices according to a pre-set program. Furthermore, touch panels can be implemented using various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch panel, input unit 804 may also include other input devices. Specifically, other input devices may include, but are not limited to, one or more of function keys (such as volume control buttons, power buttons, etc.), trackballs, joysticks, etc.

[0141] Electronic devices may also include at least one sensor 805, such as a magnetometer, gyroscope, motion sensor, and other sensors. Specifically, a magnetometer is used to determine the orientation of the electronic device, and a gyroscope can be used to determine the motion posture of the electronic device, which can be used for image stabilization, navigation, and motion-sensing game scenarios. As a type of motion sensor, an accelerometer can detect the magnitude of acceleration in various directions and, when stationary, can detect the magnitude and direction of gravity, which can be used for applications that identify the posture of the electronic device, such as landscape / portrait switching, related games, and magnetometer posture calibration. Other sensors that may be configured in electronic devices, such as pressure gauges, barometers, hygrometers, thermometers, and infrared sensors, will not be elaborated here.

[0142] The audio circuit 806 may include a speaker and a microphone, providing an audio interface between the user and the electronic device. The audio circuit 806 can convert received audio data into electrical signals and transmit them to the speaker, where the speaker converts them into sound signals for output. Conversely, the microphone converts collected sound signals into electrical signals, which are then received by the audio circuit 806, converted back into audio data, and output to the processor 801 for processing. The processed audio data is then transmitted via a video circuit to, for example, another device, or output to the memory 802 for further processing.

[0143] Although not shown, the electronic device may also include a power supply and a camera. Optionally, the camera may be positioned in the front or rear of the electronic device, and this application embodiment does not limit this.

[0144] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device. In other embodiments of this application, the electronic device may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0145] In this embodiment of the application, the processor 801 also has the following functions: Extract the data characteristics of sensor data and the signal characteristics of wireless communication signals. The sensor data is the data collected by the physical sensors of the first device. By fusing data features and signal features, a fused feature is obtained; Identify the current scene based on fused features.

[0146] In this embodiment of the application, the processor 801 also has the following functions: Search the scene database for target reference features that match the fused features. The scene database includes at least one reference feature and at least one scene, and one scene corresponds to one or more reference features. The scene corresponding to the target reference feature is determined as the current scene.

[0147] In this embodiment, the scene database also includes context information associated with each scene, which includes time information and / or location information corresponding to each scene; the processor 801 also has the following functions: Match the target context information associated with the scene corresponding to the target reference feature with the current context information; When the target context information matches the current context information, the scene corresponding to the target reference feature is determined as the current scene.

[0148] In this embodiment of the application, the processor 801 also has the following functions: The scene recognition model identifies the current scene based on fused features and current context information, which includes current time information and / or location information.

[0149] In this embodiment of the application, the processor 801 also has the following functions: Get the current battery level of the first device; When the current power level is less than the first power threshold, sensor data is collected by the target physical sensor, and the power consumption of the target physical sensor per unit time is less than or equal to the second power threshold.

[0150] In this embodiment of the application, the processor 801 also has the following functions: Obtain the user's historical behavior in the current scenario; Push content related to historical behavior.

[0151] In this embodiment of the application, the processor 801 also has the following functions: Obtain the frequency of each historical behavior among multiple historical behaviors; Based on the frequency of each historical behavior, determine the priority of pushing content related to each historical behavior; Based on the push priority of content related to each historical behavior, push at least one type of content related to the historical behavior.

[0152] In this application embodiment, the push methods for content related to historical behavior include notification bar push and / or silent push.

[0153] In this embodiment, the content related to historical behavior includes content related to the first historical behavior. The push method for the content related to the first historical behavior is silent push. The processor 801 also has the following functions: If the status information of the first device meets the target conditions, silently push content related to the first historical behavior. The status information includes at least one of the following: battery level, network speed, CPU utilization, foreground applications, and background applications.

[0154] In this embodiment of the application, the processor 801 also has the following functions: If the current scenario differs from the target scenario, an alarm message is sent to the second device.

[0155] This application discloses a computer-readable storage medium storing executable program code thereon. When the executable program code is executed by a processor, it implements the method executed by the electronic device in this application.

[0156] This application discloses a computer program product that, when run on a computer, enables the computer to implement the method executed by the electronic device in this application.

[0157] This application discloses an application publishing platform for publishing computer program products. When the computer program product is run on a computer, the computer implements the method executed by the electronic device in this application.

[0158] It should be noted that the descriptions of the storage medium and device embodiments above are similar to those of the method embodiments above, and have similar beneficial effects. For technical details not disclosed in the storage medium, storage medium, and device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0159] It should be understood that the phrases "one embodiment," "an embodiment," or "some embodiments" throughout the specification mean that a specific feature, structure, or characteristic related to an embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment," "in one embodiment," or "in some embodiments" appearing throughout the specification do not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely descriptive and do not represent the superiority or inferiority of the embodiments. The descriptions of the various embodiments above tend to emphasize the differences between the various embodiments; their similarities or commonalities can be referred to mutually, and for the sake of brevity, they will not be repeated here.

[0160] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three kinds of relationships. For example, object A and / or object B can represent three situations: object A exists alone, object A and object B exist simultaneously, and object B exists alone.

[0161] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0162] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple modules or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or modules can be electrical, mechanical, or other forms.

[0163] The modules described above as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules. They may be located in one place or distributed across multiple network units. Some or all of the modules may be selected to achieve the purpose of this embodiment according to actual needs.

[0164] In addition, each functional module in the various embodiments of this application can be integrated into one processing unit, or each module can be a separate unit, or two or more modules can be integrated into one unit; the integrated modules can be implemented in hardware or in the form of hardware plus software functional units.

[0165] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0166] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0167] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0168] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0169] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0170] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A scene recognition method, characterized in that, The method is applicable to a first device and includes: Extract the data features of the sensor data and the signal features of the wireless communication signal, wherein the sensor data is the data collected by the physical sensors of the first device; The data features and the signal features are fused to obtain the fused features; The current scene is identified based on the fused features.

2. The method according to claim 1, characterized in that, The step of identifying the current scene based on the fused features includes: Search a scene database for a target reference feature that matches the fused feature. The scene database includes at least one reference feature and at least one scene, with one scene corresponding to one or more reference features. The scene corresponding to the target reference feature is determined as the current scene.

3. The method according to claim 2, characterized in that, The scene database also includes context information associated with each scene, the context information including time information and / or location information corresponding to each scene; determining the scene corresponding to the target reference feature as the current scene includes: The target context information associated with the scene corresponding to the target reference feature is matched with the current context information; If the target context information matches the current context information, the scene corresponding to the target reference feature is determined as the current scene.

4. The method according to claim 1, characterized in that, The step of identifying the current scene based on the fused features includes: The scene recognition model identifies the current scene based on the fused features and current context information, which includes current time information and / or location information.

5. The method according to claim 1, characterized in that, Before extracting the data features of the sensor data and the signal features of the wireless communication signal, the method further includes: Obtain the current battery level of the first device; When the current power level is less than a first power threshold, sensor data is collected by a target physical sensor, and the power consumption of the target physical sensor per unit time is less than or equal to a second power threshold.

6. The method according to any one of claims 1-5, characterized in that, After identifying the current scene based on the fused features, the method further includes: Obtain the user's historical behavior in the current scenario; Push content related to the aforementioned historical behavior.

7. The method according to claim 6, characterized in that, The content pushed in relation to the historical behavior includes: Obtain the frequency of each historical behavior among multiple historical behaviors; Based on the frequency of each historical behavior, the priority of pushing content related to each historical behavior is determined; Based on the push priority of content related to each historical behavior, push at least one type of content related to the historical behavior.

8. The method according to claim 6, characterized in that, The content is pushed via notification bar push and / or silent push.

9. The method according to claim 6, characterized in that, The content related to the historical behavior includes content related to a first historical behavior, and the push method for the content related to the first historical behavior is silent push. The push of content related to the historical behavior includes: When the status information of the first device meets the target conditions, content related to the first historical behavior is silently pushed. The status information includes at least one of the following: battery level, network speed, CPU utilization, foreground applications, and background applications.

10. The method according to claim 1, characterized in that, After identifying the current scene based on the fused features, the method further includes: If the current scene differs from the target scene, an alarm message is sent to the second device.

11. A scene recognition device, characterized in that, The device is adapted for a first apparatus, the device comprising: A feature extraction unit is used to extract data features of sensor data and signal features of wireless communication signals, wherein the sensor data is data collected by the physical sensors of the first device; A feature fusion unit is used to fuse the data features and the signal features to obtain fused features; A scene recognition unit is used to identify the current scene based on the fused features.

12. An electronic device, characterized in that, include: Memory containing executable program code; and the processor coupled to the memory; The processor calls the executable program code stored in the memory, and when the executable program code is executed by the processor, the processor implements the method as described in any one of claims 1-10.

13. A computer-readable storage medium having executable program code stored thereon, characterized in that, When the executable program code is executed by the processor, it implements the method as described in any one of claims 1-10.

Images