A user monitoring method and system based on smart building
By extracting and analyzing the action set of building monitoring objects, and using monitoring objects to analyze network adjustment feature representations, the problem of low user monitoring reliability in the prior art is solved, and higher user monitoring reliability and information richness are achieved.
Patent Information
- Application Number
- CN202311149893.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-09-07
- Publication Date
- 2024-12-24
- Estimated Expiration
- 2043-09-07
AI Technical Summary
In the existing smart building user monitoring technology, the reliability of user feature information analysis is not high, resulting in low reliability of user monitoring.
By extracting the action set of building monitoring objects, forming a data group to be processed, and using the monitoring object analysis network for cyclical updates, calculating the value of network learning agents to adjust the representation of object information characteristics, and determining the target object feature information when determining the preset conditions to be met for users monitor.
It improves the reliability of user monitoring, ensures the reliability and richness of target object feature information, and enhances the accuracy of monitoring operations.
Smart Images
Figure CN117115741B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data processing technology, and in particular to a user monitoring method and system based on a smart building. Background Art
[0002] Among the many applications of smart buildings, image acquisition equipment deployed based on smart buildings is generally used to monitor building users, that is, to acquire images. Then, action analysis can be performed through the acquired user images to obtain corresponding user feature information, thereby performing user monitoring operations based on the user feature information, such as analyzing whether the user has illegal actions or illegal tendencies, and performing user management, etc. However, in the prior art, since the reliability of the analysis of user feature information is not high, the reliability of user monitoring is also not high. Summary of the invention
[0003] In view of this, an object of the present invention is to provide a user monitoring method and system based on a smart building to improve the reliability of user monitoring.
[0004] To achieve the above objectives, the embodiments of the present invention adopt the following technical solutions:
[0005] A user monitoring method based on a smart building, comprising:
[0006] Extracting a monitoring object action set of a first building monitoring object and a plurality of second building monitoring objects, wherein for each of the monitoring object action sets, the monitoring object action set includes a plurality of monitoring object actions of a building monitoring object occurring in a target building area;
[0007] Based on each of the monitoring object action sets, a corresponding to-be-processed data group is formed by combining, for each to-be-processed data included in the to-be-processed data group, the to-be-processed data includes a first action cluster formed based on at least two monitoring object actions in the monitoring object action set of the first building monitoring object, and a second action cluster formed based on at least two monitoring object actions in the monitoring object action set of the second building monitoring object, and in the to-be-processed data, the first action cluster and the second action cluster have the same action;
[0008] Based on the data group to be processed, the monitoring object analysis network is cyclically updated. In each updating process, a plurality of network learning cost values are calculated based on the object information feature representation of the first building monitoring object, the first action cluster and the second action cluster included in each of the data to be processed, each of the network learning cost values is used to reflect the difference between the first building monitoring object and the second building monitoring object, and the object information feature representation is adjusted based on the plurality of network learning cost values;
[0009] If it is determined based on the multiple network learning cost values that the monitoring object analysis network meets the preset conditions, then based on the current object information feature representation, the target object feature information corresponding to the first building monitoring object is determined, and user monitoring operations are performed on the first building monitoring object based on the target object feature information.
[0010] In some preferred embodiments, in the above-mentioned user monitoring method based on smart buildings, the step of extracting a monitoring object action set of a first building monitoring object and a plurality of second building monitoring objects includes:
[0011] Monitoring a target building area to form a building monitoring image set, wherein the building monitoring image set includes a plurality of frames of building monitoring images;
[0012] Performing object recognition processing on each of the multiple building monitoring images to output each building monitoring object in each building monitoring image, and forming an initial set of building monitoring objects based on each building monitoring object in each building monitoring image, and performing deduplication processing on the building monitoring objects in the initial set of building monitoring objects to form a corresponding building monitoring object target set;
[0013] Determine any one of the building monitoring objects in the building monitoring object target set as the first building monitoring object, and determine each building monitoring object other than the first building monitoring object in the building monitoring object target set as the second building monitoring object; or, determine the building monitoring object with the largest number of frames of the corresponding building monitoring image in the building monitoring object target set as the first building monitoring object, and determine each building monitoring object other than the first building monitoring object in the building monitoring object target set as the second building monitoring object;
[0014] In the building monitoring image set, each frame of the building monitoring image corresponding to the first building monitoring object is extracted, and classified and combined based on the continuity between the building monitoring images to form at least one first building monitoring image subset, and action recognition processing of the first building monitoring object is performed on each of the first building monitoring image subsets to form a monitoring object action corresponding to each of the first building monitoring image subsets, and then based on the monitoring object action corresponding to each of the first building monitoring image subsets, a monitoring object action set of the first building monitoring object is combined;
[0015] In the building surveillance image set, each frame of building surveillance image corresponding to each of the second building surveillance objects is extracted, and classified and combined based on the continuity between the building surveillance images to form at least one second building surveillance image subset corresponding to each of the second building surveillance objects, and action recognition processing of the second building surveillance object is performed on each of the second building surveillance image subsets to form a surveillance object action corresponding to each second building surveillance image subset, and then based on the surveillance object action corresponding to each of the first building surveillance image subsets, a surveillance object action set for each second building surveillance object is combined.
[0016] In some preferred embodiments, in the above-mentioned user monitoring method based on smart buildings, the first action cluster includes at least two adjacent monitoring object actions in the monitoring object action set of the first building monitoring object, and the second action cluster includes at least two adjacent monitoring object actions in the monitoring object action set of the second building monitoring object.
[0017] In some preferred embodiments, in the above-mentioned user monitoring method based on smart buildings, for each of the data to be processed, the same action part of the first action cluster and the second action cluster, the different action part of the first action cluster and the same action part, and the different action part of the second action cluster and the same action part in the data to be processed are taken as a target data group;
[0018] The step of calculating a plurality of network learning cost values corresponding to the first action cluster and the second action cluster included in each of the data to be processed based on the object information feature representation of the first building monitoring object comprises:
[0019] Based on the object information feature representation of the first building monitoring object and the target data group corresponding to each of the to-be-processed data, a network learning cost value of the data group corresponding to each of the to-be-processed data is determined.
[0020] In some preferred embodiments, in the above-mentioned user monitoring method based on smart buildings, the step of calculating a plurality of network learning cost values corresponding to the first action cluster and the second action cluster included in each of the data to be processed based on the object information feature representation of the first building monitoring object includes:
[0021] For each piece of data to be processed, based on the object information feature representation and the action information feature representation of the monitoring object action in each piece of data to be processed, determining that the monitoring object action set of the first building monitoring object includes a first possibility evaluation value of the first action cluster and a second possibility evaluation value of the second action cluster;
[0022] Based on the first possibility evaluation value and the second possibility evaluation value, a network learning cost value of each of the to-be-processed data is determined, and the network learning cost value is positively correlated with the first possibility evaluation value and negatively correlated with the second possibility evaluation value.
[0023] In some preferred embodiments, in the above-mentioned user monitoring method based on smart buildings, the first action cluster includes a first monitoring object action and a second monitoring object action, and the second action cluster includes the second monitoring object action and a third monitoring object action;
[0024] The step of determining, for each piece of data to be processed, based on the object information feature representation and the action information feature representation of the monitoring object action in each piece of data to be processed, that the monitoring object action set of the first building monitoring object includes a first possibility evaluation value of the first action cluster and a second possibility evaluation value of the second action cluster, comprises:
[0025] Determine a first matching parameter of the first monitored object action and the second monitored object action based on the object information feature representation, wherein the first matching parameter is used to reflect the degree of matching between the action information feature representation of the first monitored object action and the action information feature representation of the second monitored object action and the object information feature representation;
[0026] Determine a second matching parameter of the second monitored object action and the third monitored object action based on the object information feature representation, wherein the second matching parameter is used to reflect the degree of matching between the action information feature representation of the second monitored object action and the action information feature representation of the third monitored object action and the object information feature representation;
[0027] Determining a first likelihood evaluation value based on the first matching parameter, wherein the first matching parameter is positively correlated with the first likelihood evaluation value;
[0028] Based on the second matching parameter, a second likelihood evaluation value is determined, the second matching parameter being positively correlated to the second likelihood evaluation value.
[0029] In some preferred embodiments, in the above-mentioned user monitoring method based on smart buildings, before the step of performing a cyclic update operation on the monitoring object analysis network based on the data group to be processed, the user monitoring method based on smart buildings further includes:
[0030] Based on the action representative monitoring image of each monitored object action, determine the action image feature representation set of each monitored object action respectively;
[0031] Based on the action image feature representation set of each monitored object action, the action information feature representation of each monitored object action is determined respectively.
[0032] The embodiment of the present invention further provides a user monitoring system based on a smart building, comprising:
[0033] An object action extraction module is used to extract a monitoring object action set of a first building monitoring object and a plurality of second building monitoring objects, wherein for each monitoring object action set, the monitoring object action set includes a plurality of monitoring object actions of a building monitoring object occurring in a target building area;
[0034] A data group forming module, used for combining and forming a corresponding to-be-processed data group based on each of the monitoring object action sets, wherein for each to-be-processed data included in the to-be-processed data group, the to-be-processed data includes a first action cluster formed based on at least two monitoring object actions in the monitoring object action set of the first building monitoring object, and a second action cluster formed based on at least two monitoring object actions in the monitoring object action set of the second building monitoring object, wherein the first action cluster and the second action cluster have the same action in the to-be-processed data;
[0035] A network update module, for performing a cyclic update operation on the monitoring object analysis network based on the data group to be processed, and in each updating process, calculating a plurality of network learning cost values based on the object information feature representation of the first building monitoring object, the first action cluster and the second action cluster included in each of the data to be processed, each of the network learning cost values being used to reflect the difference between the first building monitoring object and the second building monitoring object, and adjusting the object information feature representation based on the plurality of network learning cost values;
[0036] A user monitoring module is used to determine the target object feature information corresponding to the first building monitoring object based on the current feature representation of the object information if it is determined that the monitoring object analysis network meets the preset conditions based on the multiple network learning cost values, and to perform user monitoring operations on the first building monitoring object based on the target object feature information.
[0037] In some preferred embodiments, in the above-mentioned user monitoring system based on smart buildings, the object action extraction module is specifically used for:
[0038] Monitoring a target building area to form a building monitoring image set, wherein the building monitoring image set includes a plurality of frames of building monitoring images;
[0039] Performing object recognition processing on each of the multiple building monitoring images to output each building monitoring object in each building monitoring image, and forming an initial set of building monitoring objects based on each building monitoring object in each building monitoring image, and performing deduplication processing on the building monitoring objects in the initial set of building monitoring objects to form a corresponding building monitoring object target set;
[0040] Determine any one of the building monitoring objects in the building monitoring object target set as the first building monitoring object, and determine each building monitoring object other than the first building monitoring object in the building monitoring object target set as the second building monitoring object; or, determine the building monitoring object with the largest number of frames of the corresponding building monitoring image in the building monitoring object target set as the first building monitoring object, and determine each building monitoring object other than the first building monitoring object in the building monitoring object target set as the second building monitoring object;
[0041] In the building monitoring image set, each frame of the building monitoring image corresponding to the first building monitoring object is extracted, and classified and combined based on the continuity between the building monitoring images to form at least one first building monitoring image subset, and action recognition processing of the first building monitoring object is performed on each of the first building monitoring image subsets to form a monitoring object action corresponding to each of the first building monitoring image subsets, and then based on the monitoring object action corresponding to each of the first building monitoring image subsets, a monitoring object action set of the first building monitoring object is combined;
[0042] In the building surveillance image set, each frame of building surveillance image corresponding to each of the second building surveillance objects is extracted, and classified and combined based on the continuity between the building surveillance images to form at least one second building surveillance image subset corresponding to each of the second building surveillance objects, and action recognition processing of the second building surveillance object is performed on each of the second building surveillance image subsets to form a surveillance object action corresponding to each second building surveillance image subset, and then based on the surveillance object action corresponding to each of the first building surveillance image subsets, a surveillance object action set for each second building surveillance object is combined.
[0043] In some preferred embodiments, in the above-mentioned user monitoring system based on smart buildings, the network update module is specifically used for:
[0044] For each piece of data to be processed, based on the object information feature representation and the action information feature representation of the monitoring object action in each piece of data to be processed, determining that the monitoring object action set of the first building monitoring object includes a first possibility evaluation value of the first action cluster and a second possibility evaluation value of the second action cluster;
[0045] Based on the first possibility evaluation value and the second possibility evaluation value, a network learning cost value of each of the to-be-processed data is determined, and the network learning cost value is positively correlated with the first possibility evaluation value and negatively correlated with the second possibility evaluation value.
[0046] A user monitoring method and system based on a smart building provided by an embodiment of the present invention can extract a monitoring object action set of a first building monitoring object and multiple second building monitoring objects; based on each monitoring object action set, a data group to be processed is formed; based on the data group to be processed, a monitoring object analysis network is cyclically updated, and during the updating process, a network learning cost value is calculated based on the object information feature representation of the first building monitoring object and the first action cluster and the second action cluster included in the data to be processed, and the object information feature representation is adjusted based on the network learning cost value; if it is determined that the monitoring object analysis network meets the preset conditions, the target object feature information is determined based on the current object information feature representation, and the user monitoring operation is performed based on the target object feature information. Based on the aforementioned scheme, on the one hand, since the target object characteristic information is determined based on the current object information characteristic representation when the monitoring object analysis network meets the preset conditions, the reliability of the user monitoring operation based on the target object characteristic information is higher; on the other hand, since the object information characteristic representation of the first building monitoring object is adjusted based on the monitoring object action set of the first building monitoring object and multiple second building monitoring objects, the obtained target object characteristic information is more reliable and has richer information, thereby effectively improving the reliability of user monitoring.
[0047] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] Figure 1 A structural block diagram of a user monitoring platform based on smart buildings provided in an embodiment of the present invention.
[0049] Figure 2 A flowchart of the steps of the user monitoring method based on smart buildings provided in an embodiment of the present invention.
[0050] Figure 3 A schematic diagram of various modules included in the user monitoring system based on smart buildings provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0051] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.
[0052] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0053] like Figure 1 As shown, an embodiment of the present invention provides a user monitoring platform based on a smart building, wherein the user monitoring platform based on a smart building may include a memory and a processor.
[0054] In detail, the memory and the processor are electrically connected directly or indirectly to achieve data transmission or interaction. For example, they can be electrically connected to each other through one or more communication buses or signal lines. The memory can store at least one software function module (computer program) that can exist in the form of software or firmware. The processor can be used to execute the executable computer program stored in the memory, thereby realizing the user monitoring method based on smart buildings provided by the embodiment of the present invention (as described later).
[0055] Optionally, in some embodiments, the memory may be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), etc.
[0056] Optionally, in some embodiments, the processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), a system on chip (SoC), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
[0057] Optionally, in some embodiments, in an alternative example, the smart building-based user monitoring platform may be a server with data processing capabilities.
[0058] Combination Figure 2 The embodiment of the present invention also provides a user monitoring method based on smart buildings, which can be applied to the user monitoring platform based on smart buildings. The method steps defined in the process related to the user monitoring method based on smart buildings can be implemented by the user monitoring platform based on smart buildings. Figure 2 The specific process shown is explained in detail.
[0059] Step S110: extracting a monitoring object action set of a first building monitoring object and a plurality of second building monitoring objects.
[0060] In an embodiment of the present invention, the user monitoring platform based on smart buildings can extract a monitoring object action set of a first building monitoring object and multiple second building monitoring objects. For each of the monitoring object action sets, the monitoring object action set includes multiple monitoring object actions of a building monitoring object appearing in a target building area.
[0061] Step S120: Based on each of the monitored object action sets, a corresponding data group to be processed is formed.
[0062] In an embodiment of the present invention, the user monitoring platform based on the smart building can be combined to form a corresponding data group to be processed based on each of the monitoring object action sets. For each data to be processed included in the data group to be processed, the data to be processed includes a first action cluster (such as action 1, action 2, action 3) formed based on at least two monitoring object actions in the monitoring object action set of the first building monitoring object, and a second action cluster (such as action 2, action 4, action 5) formed based on at least two monitoring object actions in the monitoring object action set of the second building monitoring object, and in the data to be processed, the first action cluster and the second action cluster have the same action (such as action 2 in the aforementioned example).
[0063] Step S130: based on the data group to be processed, a monitoring object analysis network is cyclically updated.
[0064] In an embodiment of the present invention, the user monitoring platform based on the smart building can perform a cyclic update operation on the monitoring object analysis network based on the data group to be processed. In each update process, multiple network learning cost values are calculated based on the object information feature representation of the first building monitoring object, the first action cluster and the second action cluster included in each of the data to be processed, each of the network learning cost values is used to reflect the difference between the first building monitoring object and the second building monitoring object, and the object information feature representation is adjusted based on the multiple network learning cost values (so that the network learning cost value is reduced).
[0065] Step S140: If it is determined that the monitoring object analysis network meets the preset conditions based on the multiple network learning cost values, the target object feature information corresponding to the first building monitoring object is determined based on the current object information feature representation, and user monitoring operations are performed on the first building monitoring object based on the target object feature information.
[0066] In an embodiment of the present invention, the user monitoring platform based on a smart building can determine the target object feature information corresponding to the first building monitoring object based on the current object information feature representation (for example, the current object information feature representation can be used as the target object feature information) when it is determined that the monitoring object analysis network meets the preset conditions based on the multiple network learning cost values (such as the network learning cost value converges or the number of updates is greater than the preset value), and perform user monitoring operations on the first building monitoring object based on the target object feature information, for example, performing abnormal analysis of user actions based on the target object feature information, such as analyzing whether it is abnormal or the degree of abnormality, etc., which can be achieved based on the corresponding neural network.
[0067] Based on the aforementioned scheme, on the one hand, since the target object characteristic information is determined based on the current object information characteristic representation when the monitoring object analysis network meets the preset conditions, the reliability of the user monitoring operation based on the target object characteristic information is higher; on the other hand, since the object information characteristic representation of the first building monitoring object is adjusted based on the monitoring object action set of the first building monitoring object and multiple second building monitoring objects, the obtained target object characteristic information is more reliable and has richer information, thereby effectively improving the reliability of user monitoring.
[0068] Optionally, in some implementations, the step of extracting a monitoring object action set of the first building monitoring object and a plurality of second building monitoring objects may further include the following content:
[0069] Monitoring a target building area to form a building monitoring image set, wherein the building monitoring image set includes a plurality of frames of building monitoring images;
[0070] Performing object recognition processing on each of the multiple building monitoring images to output each building monitoring object in each building monitoring image, and forming an initial set of building monitoring objects based on each building monitoring object in each building monitoring image, and performing deduplication processing on the building monitoring objects in the initial set of building monitoring objects (retaining one identical building monitoring object) to form a corresponding building monitoring object target set;
[0071] Determine any one of the building monitoring objects in the building monitoring object target set as the first building monitoring object, and determine each building monitoring object other than the first building monitoring object in the building monitoring object target set as the second building monitoring object; or, determine the building monitoring object with the largest number of frames of the corresponding building monitoring image in the building monitoring object target set as the first building monitoring object, and determine each building monitoring object other than the first building monitoring object in the building monitoring object target set as the second building monitoring object;
[0072] In the building surveillance image set, each frame of building surveillance image corresponding to the first building surveillance object is extracted, and classified and combined based on the continuity between the building surveillance images to form at least one first building surveillance image subset (that is, continuous building surveillance images are combined to form a first building surveillance image subset), and action recognition processing of the first building surveillance object is performed on each of the first building surveillance image subsets (that is, for the partial image area corresponding to the first building surveillance object) to form a surveillance object action corresponding to each of the first building surveillance image subsets, and then based on the surveillance object action corresponding to each of the first building surveillance image subsets, a surveillance object action set of the first building surveillance object is combined;
[0073] In the building surveillance image set, each frame of building surveillance image corresponding to each of the second building surveillance objects is extracted, and classified and combined based on the continuity between the building surveillance images to form at least one second building surveillance image subset corresponding to each of the second building surveillance objects, and action recognition processing of the second building surveillance object is performed on each of the second building surveillance image subsets to form a surveillance object action corresponding to each second building surveillance image subset, and then based on the surveillance object action corresponding to each of the first building surveillance image subsets, a surveillance object action set for each second building surveillance object is combined.
[0074] Optionally, in some embodiments, the first action cluster includes at least two adjacent monitoring object actions in the monitoring object action set of the first building monitoring object, and the second action cluster includes at least two adjacent monitoring object actions in the monitoring object action set of the second building monitoring object.
[0075] Optionally, in some embodiments, for each of the data to be processed, the same action part (such as action 2) of the first action cluster (such as action 1, action 2) and the second action cluster (such as action 2, action 3) in the data to be processed, the different action part (such as action 1) between the first action cluster and the same action part, and the different action part (such as action 3) between the second action cluster and the same action part are taken as a target data group. Based on this, the step of calculating a plurality of network learning cost values corresponding to the first action cluster and the second action cluster included in each of the data to be processed based on the object information feature representation of the first building monitoring object may further include the following contents:
[0076] Based on the object information feature representation of the first building monitoring object and the target data group corresponding to each of the to-be-processed data, a network learning cost value (ie, Triplet loss) of the data group corresponding to each of the to-be-processed data is determined.
[0077] Optionally, in some implementations, the step of calculating a plurality of network learning cost values corresponding to the first action cluster and the second action cluster included in each of the data to be processed based on the object information feature representation of the first building monitoring object may further include the following contents:
[0078] For each piece of data to be processed, based on the object information feature representation and the action information feature representation of the monitoring object action in each piece of data to be processed, determining that the monitoring object action set of the first building monitoring object includes a first possibility evaluation value of the first action cluster and a second possibility evaluation value of the second action cluster;
[0079] Based on the first possibility evaluation value and the second possibility evaluation value, a network learning cost value of each of the to-be-processed data is determined, and the network learning cost value is positively correlated with the first possibility evaluation value and negatively correlated with the second possibility evaluation value.
[0080] Alternatively, in some embodiments, the first action cluster includes a first monitoring object action and a second monitoring object action, and the second action cluster includes the second monitoring object action and a third monitoring object action. Based on this, for each data to be processed, based on the object information feature representation and the action information feature representation of the monitoring object action in each data to be processed, the step of determining that the monitoring object action set of the first building monitoring object includes the first possibility evaluation value of the first action cluster and the second possibility evaluation value of the second action cluster may further include the following contents:
[0081] Determining a first matching parameter of the first monitored object action and the second monitored object action based on the object information feature representation, wherein the first matching parameter is used to reflect the degree of matching between the action information feature representation of the first monitored object action and the action information feature representation of the second monitored object action and the object information feature representation (for example, the action information feature representation of the first monitored object action and the action information feature representation of the second monitored object action may be spliced to form a first spliced feature representation, and then the similarity between the first spliced feature representation and the object information feature representation may be calculated to obtain a first matching parameter);
[0082] Determining a second matching parameter of the second monitored object action and the third monitored object action based on the object information feature representation, where the second matching parameter is used to reflect the degree of matching between the action information feature representation of the second monitored object action and the action information feature representation of the third monitored object action and the object information feature representation (for example, the action information feature representation of the second monitored object action and the action information feature representation of the third monitored object action may be spliced to form a second spliced feature representation, and then the similarity between the second spliced feature representation and the object information feature representation may be calculated to obtain a second matching parameter);
[0083] Determining a first likelihood evaluation value based on the first matching parameter, wherein the first matching parameter is positively correlated with the first likelihood evaluation value;
[0084] Based on the second matching parameter, a second likelihood evaluation value is determined, the second matching parameter being positively correlated to the second likelihood evaluation value.
[0085] Optionally, in some embodiments, before the step of performing a cyclic update operation on the monitoring object analysis network based on the to-be-processed data group, the user monitoring method based on the smart building may further include the following contents:
[0086] Based on the action representative monitoring image of each monitoring object action (such as each corresponding building monitoring image frame), determine the action image feature representation set of each monitoring object action (for a monitoring object action, feature mining can be performed on each building monitoring image frame corresponding to the monitoring object action to obtain the action image feature representation corresponding to each building monitoring image frame, so that the corresponding action image feature representation set can be combined);
[0087] Based on the action image feature representation set of each monitored object action, the action information feature representation of each monitored object action is determined respectively (for example, for an action image feature representation set, inter-modal focus feature analysis can be performed on every two action image feature representations in the action image feature representation set, and then the obtained focus feature analysis results are spliced to form a corresponding action information feature representation of a monitored object action).
[0088] Combination Figure 3 The embodiment of the present invention further provides a user monitoring system based on a smart building, which can be applied to the user monitoring platform based on a smart building. The user monitoring system based on a smart building can include the following contents:
[0089] An object action extraction module is used to extract a monitoring object action set of a first building monitoring object and a plurality of second building monitoring objects, wherein for each monitoring object action set, the monitoring object action set includes a plurality of monitoring object actions of a building monitoring object occurring in a target building area;
[0090] A data group forming module, used for combining and forming a corresponding to-be-processed data group based on each of the monitoring object action sets, wherein for each to-be-processed data included in the to-be-processed data group, the to-be-processed data includes a first action cluster formed based on at least two monitoring object actions in the monitoring object action set of the first building monitoring object, and a second action cluster formed based on at least two monitoring object actions in the monitoring object action set of the second building monitoring object, wherein the first action cluster and the second action cluster have the same action in the to-be-processed data;
[0091] A network update module, for performing a cyclic update operation on the monitoring object analysis network based on the data group to be processed, and in each updating process, calculating a plurality of network learning cost values based on the object information feature representation of the first building monitoring object, the first action cluster and the second action cluster included in each of the data to be processed, each of the network learning cost values being used to reflect the difference between the first building monitoring object and the second building monitoring object, and adjusting the object information feature representation based on the plurality of network learning cost values;
[0092] A user monitoring module is used to determine the target object feature information corresponding to the first building monitoring object based on the current feature representation of the object information if it is determined that the monitoring object analysis network meets the preset conditions based on the multiple network learning cost values, and to perform user monitoring operations on the first building monitoring object based on the target object feature information.
[0093] Optionally, in some implementations, the object action extraction module is specifically used to:
[0094] Monitoring a target building area to form a building monitoring image set, wherein the building monitoring image set includes a plurality of frames of building monitoring images;
[0095] Performing object recognition processing on each of the multiple building monitoring images to output each building monitoring object in each building monitoring image, and forming an initial set of building monitoring objects based on each building monitoring object in each building monitoring image, and performing deduplication processing on the building monitoring objects in the initial set of building monitoring objects to form a corresponding building monitoring object target set;
[0096] Determine any one of the building monitoring objects in the building monitoring object target set as the first building monitoring object, and determine each building monitoring object other than the first building monitoring object in the building monitoring object target set as the second building monitoring object; or, determine the building monitoring object with the largest number of frames of the corresponding building monitoring image in the building monitoring object target set as the first building monitoring object, and determine each building monitoring object other than the first building monitoring object in the building monitoring object target set as the second building monitoring object;
[0097] In the building monitoring image set, each frame of the building monitoring image corresponding to the first building monitoring object is extracted, and classified and combined based on the continuity between the building monitoring images to form at least one first building monitoring image subset, and action recognition processing of the first building monitoring object is performed on each of the first building monitoring image subsets to form a monitoring object action corresponding to each of the first building monitoring image subsets, and then based on the monitoring object action corresponding to each of the first building monitoring image subsets, a monitoring object action set of the first building monitoring object is combined;
[0098] In the building surveillance image set, each frame of building surveillance image corresponding to each of the second building surveillance objects is extracted, and classified and combined based on the continuity between the building surveillance images to form at least one second building surveillance image subset corresponding to each of the second building surveillance objects, and action recognition processing of the second building surveillance object is performed on each of the second building surveillance image subsets to form a surveillance object action corresponding to each second building surveillance image subset, and then based on the surveillance object action corresponding to each of the first building surveillance image subsets, a surveillance object action set for each second building surveillance object is combined.
[0099] Optionally, in some implementations, the network update module is specifically used to:
[0100] For each piece of data to be processed, based on the object information feature representation and the action information feature representation of the monitoring object action in each piece of data to be processed, determining that the monitoring object action set of the first building monitoring object includes a first possibility evaluation value of the first action cluster and a second possibility evaluation value of the second action cluster;
[0101] Based on the first possibility evaluation value and the second possibility evaluation value, a network learning cost value of each of the to-be-processed data is determined, and the network learning cost value is positively correlated with the first possibility evaluation value and negatively correlated with the second possibility evaluation value.
[0102] In summary, the user monitoring method and system based on smart buildings provided by the present invention can extract the monitoring object action sets of the first building monitoring object and multiple second building monitoring objects; based on each monitoring object action set, a data group to be processed is formed; based on the data group to be processed, the monitoring object analysis network is cyclically updated, and during the updating process, the network learning cost value is calculated based on the object information feature representation of the first building monitoring object and the first action cluster and the second action cluster included in the data to be processed, and the object information feature representation is adjusted based on the network learning cost value; if it is determined that the monitoring object analysis network meets the preset conditions, the target object feature information is determined based on the current object information feature representation, and the user monitoring operation is performed based on the target object feature information. Based on the aforementioned scheme, on the one hand, since the target object characteristic information is determined based on the current object information characteristic representation when the monitoring object analysis network meets the preset conditions, the reliability of the user monitoring operation based on the target object characteristic information is higher; on the other hand, since the object information characteristic representation of the first building monitoring object is adjusted based on the monitoring object action set of the first building monitoring object and multiple second building monitoring objects, the obtained target object characteristic information is more reliable and has richer information, thereby effectively improving the reliability of user monitoring.
[0103] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A user monitoring method based on smart buildings, characterized in that: include: Extracting a monitoring object action set of a first building monitoring object and a plurality of second building monitoring objects, wherein for each of the monitoring object action sets, the monitoring object action set includes a plurality of monitoring object actions of a building monitoring object occurring in a target building area; Based on each of the monitoring object action sets, a corresponding to-be-processed data group is formed by combining, for each to-be-processed data included in the to-be-processed data group, the to-be-processed data includes a first action cluster formed based on at least two monitoring object actions in the monitoring object action set of the first building monitoring object, and a second action cluster formed based on at least two monitoring object actions in the monitoring object action set of the second building monitoring object, and in the to-be-processed data, the first action cluster and the second action cluster have the same action; Based on the data group to be processed, the monitoring object analysis network is cyclically updated. In each updating process, a plurality of network learning cost values are calculated based on the object information feature representation of the first building monitoring object, the first action cluster and the second action cluster included in each of the data to be processed, each of the network learning cost values is used to reflect the difference between the first building monitoring object and the second building monitoring object, and the object information feature representation is adjusted based on the plurality of network learning cost values; If it is determined based on the multiple network learning cost values that the monitoring object analysis network meets the preset conditions, then based on the current object information feature representation, the target object feature information corresponding to the first building monitoring object is determined, and user monitoring operations are performed on the first building monitoring object based on the target object feature information.
2. The user monitoring method based on smart buildings according to claim 1, characterized in that: The step of extracting a monitoring object action set of a first building monitoring object and a plurality of second building monitoring objects comprises: Monitoring a target building area to form a building monitoring image set, wherein the building monitoring image set includes a plurality of frames of building monitoring images; Performing object recognition processing on each of the multiple building monitoring images to output each building monitoring object in each building monitoring image, and forming an initial set of building monitoring objects based on each building monitoring object in each building monitoring image, and performing deduplication processing on the building monitoring objects in the initial set of building monitoring objects to form a corresponding building monitoring object target set; Determine any one of the building monitoring objects in the building monitoring object target set as the first building monitoring object, and determine each building monitoring object other than the first building monitoring object in the building monitoring object target set as the second building monitoring object; or, determine the building monitoring object with the largest number of frames of the corresponding building monitoring image in the building monitoring object target set as the first building monitoring object, and determine each building monitoring object other than the first building monitoring object in the building monitoring object target set as the second building monitoring object; In the building monitoring image set, each frame of the building monitoring image corresponding to the first building monitoring object is extracted, and classified and combined based on the continuity between the building monitoring images to form at least one first building monitoring image subset, and action recognition processing of the first building monitoring object is performed on each of the first building monitoring image subsets to form a monitoring object action corresponding to each of the first building monitoring image subsets, and then based on the monitoring object action corresponding to each of the first building monitoring image subsets, a monitoring object action set of the first building monitoring object is combined; In the building surveillance image set, each frame of building surveillance image corresponding to each of the second building surveillance objects is extracted, and classified and combined based on the continuity between the building surveillance images to form at least one second building surveillance image subset corresponding to each of the second building surveillance objects, and action recognition processing of the second building surveillance object is performed on each of the second building surveillance image subsets to form a surveillance object action corresponding to each second building surveillance image subset, and then based on the surveillance object action corresponding to each of the first building surveillance image subsets, a surveillance object action set for each second building surveillance object is combined.
3. The user monitoring method based on smart building according to claim 1, characterized in that: The first action cluster includes at least two adjacent monitoring object actions in the monitoring object action set of the first building monitoring object, and the second action cluster includes at least two adjacent monitoring object actions in the monitoring object action set of the second building monitoring object.
4. The user monitoring method based on smart buildings as claimed in claim 1, characterized in that: For each of the to-be-processed data, the same action part of the first action cluster and the second action cluster, the different action part of the first action cluster and the same action part, and the different action part of the second action cluster and the same action part in the to-be-processed data are taken as a target data group; The step of calculating a plurality of network learning cost values corresponding to the first action cluster and the second action cluster included in each of the data to be processed based on the object information feature representation of the first building monitoring object comprises: Based on the object information feature representation of the first building monitoring object and the target data group corresponding to each of the to-be-processed data, a network learning cost value of the data group corresponding to each of the to-be-processed data is determined.
5. The user monitoring method based on smart building according to claim 1, characterized in that: The step of calculating a plurality of network learning cost values corresponding to the first action cluster and the second action cluster included in each of the data to be processed based on the object information feature representation of the first building monitoring object comprises: For each piece of data to be processed, based on the object information feature representation and the action information feature representation of the monitoring object action in each piece of data to be processed, determining that the monitoring object action set of the first building monitoring object includes a first possibility evaluation value of the first action cluster and a second possibility evaluation value of the second action cluster; Based on the first possibility evaluation value and the second possibility evaluation value, a network learning cost value of each of the to-be-processed data is determined, and the network learning cost value is positively correlated with the first possibility evaluation value and negatively correlated with the second possibility evaluation value.
6. The user monitoring method based on smart building according to claim 5, characterized in that: The first action cluster includes a first monitoring object action and a second monitoring object action, and the second action cluster includes the second monitoring object action and a third monitoring object action; The step of determining, for each piece of data to be processed, based on the object information feature representation and the action information feature representation of the monitoring object action in each piece of data to be processed, that the monitoring object action set of the first building monitoring object includes a first possibility evaluation value of the first action cluster and a second possibility evaluation value of the second action cluster, comprises: Determine a first matching parameter of the first monitored object action and the second monitored object action based on the object information feature representation, wherein the first matching parameter is used to reflect the degree of matching between the action information feature representation of the first monitored object action and the action information feature representation of the second monitored object action and the object information feature representation; Determine a second matching parameter of the second monitored object action and the third monitored object action based on the object information feature representation, wherein the second matching parameter is used to reflect the degree of matching between the action information feature representation of the second monitored object action and the action information feature representation of the third monitored object action and the object information feature representation; Determining a first likelihood evaluation value based on the first matching parameter, wherein the first matching parameter is positively correlated with the first likelihood evaluation value; Based on the second matching parameter, a second likelihood evaluation value is determined, the second matching parameter being positively correlated to the second likelihood evaluation value.
7. The user monitoring method based on smart buildings according to any one of claims 1 to 6, characterized in that: Before the step of performing a cyclic update operation on the monitoring object analysis network based on the to-be-processed data group, the user monitoring method based on the smart building further includes: Based on the action representative monitoring image of each monitored object action, determine the action image feature representation set of each monitored object action respectively; Based on the action image feature representation set of each monitored object action, the action information feature representation of each monitored object action is determined respectively.
8. A user monitoring system based on smart buildings, characterized in that: include: An object action extraction module is used to extract a monitoring object action set of a first building monitoring object and a plurality of second building monitoring objects, wherein for each monitoring object action set, the monitoring object action set includes a plurality of monitoring object actions of a building monitoring object occurring in a target building area; A data group forming module, used for combining and forming a corresponding to-be-processed data group based on each of the monitoring object action sets, wherein for each to-be-processed data included in the to-be-processed data group, the to-be-processed data includes a first action cluster formed based on at least two monitoring object actions in the monitoring object action set of the first building monitoring object, and a second action cluster formed based on at least two monitoring object actions in the monitoring object action set of the second building monitoring object, wherein the first action cluster and the second action cluster have the same action in the to-be-processed data; A network update module, for performing a cyclic update operation on the monitoring object analysis network based on the data group to be processed, and in each updating process, calculating a plurality of network learning cost values based on the object information feature representation of the first building monitoring object, the first action cluster and the second action cluster included in each of the data to be processed, each of the network learning cost values being used to reflect the difference between the first building monitoring object and the second building monitoring object, and adjusting the object information feature representation based on the plurality of network learning cost values; A user monitoring module is used to determine the target object feature information corresponding to the first building monitoring object based on the current feature representation of the object information if it is determined that the monitoring object analysis network meets the preset conditions based on the multiple network learning cost values, and to perform user monitoring operations on the first building monitoring object based on the target object feature information.
9. The user monitoring system based on smart buildings as claimed in claim 8, characterized in that: The object action extraction module is specifically used for: Monitoring a target building area to form a building monitoring image set, wherein the building monitoring image set includes a plurality of frames of building monitoring images; Performing object recognition processing on each of the multiple building monitoring images to output each building monitoring object in each building monitoring image, and forming an initial set of building monitoring objects based on each building monitoring object in each building monitoring image, and performing deduplication processing on the building monitoring objects in the initial set of building monitoring objects to form a corresponding building monitoring object target set; Determine any one of the building monitoring objects in the building monitoring object target set as the first building monitoring object, and determine each building monitoring object other than the first building monitoring object in the building monitoring object target set as the second building monitoring object; or, determine the building monitoring object with the largest number of frames of the corresponding building monitoring image in the building monitoring object target set as the first building monitoring object, and determine each building monitoring object other than the first building monitoring object in the building monitoring object target set as the second building monitoring object; In the building monitoring image set, each frame of the building monitoring image corresponding to the first building monitoring object is extracted, and classified and combined based on the continuity between the building monitoring images to form at least one first building monitoring image subset, and action recognition processing of the first building monitoring object is performed on each of the first building monitoring image subsets to form a monitoring object action corresponding to each of the first building monitoring image subsets, and then based on the monitoring object action corresponding to each of the first building monitoring image subsets, a monitoring object action set of the first building monitoring object is combined; In the building surveillance image set, each frame of building surveillance image corresponding to each of the second building surveillance objects is extracted, and classified and combined based on the continuity between the building surveillance images to form at least one second building surveillance image subset corresponding to each of the second building surveillance objects, and action recognition processing of the second building surveillance object is performed on each of the second building surveillance image subsets to form a surveillance object action corresponding to each second building surveillance image subset, and then based on the surveillance object action corresponding to each of the first building surveillance image subsets, a surveillance object action set for each second building surveillance object is combined.
10. The user monitoring system based on smart buildings as claimed in claim 8, characterized in that: The network update module is specifically used for: For each piece of data to be processed, based on the object information feature representation and the action information feature representation of the monitoring object action in each piece of data to be processed, determining that the monitoring object action set of the first building monitoring object includes a first possibility evaluation value of the first action cluster and a second possibility evaluation value of the second action cluster; Based on the first possibility evaluation value and the second possibility evaluation value, a network learning cost value of each of the to-be-processed data is determined, and the network learning cost value is positively correlated with the first possibility evaluation value and negatively correlated with the second possibility evaluation value.
Citation Information
Patent Citations
Intelligent building data acquisition method
CN113609356A
Object classification method and device based on artificial intelligence
CN113657536A