Intelligent store conflict identification system and method

Abstract

The embodiment of the invention provides an intelligent store conflict identification system and method. The system comprises an infrared thermal imager, an audio acquisition device, an alarm system, a communication device and a processor. The thermal infrared imager is arranged in a preset area of the store, and the thermal infrared imager is configured to obtain thermal infrared image data of the preset area; the audio acquisition device is arranged in a preset area of the store, and the audio acquisition device is configured to acquire audio data of the preset area; the alarm system is configured to send conflict information to a responsible shop assistant and execute alarm based on the alarm instruction; the communication device is configured to realize data communication among the thermal infrared imager, the audio acquisition equipment, the alarm system and the processor; the processor is configured to determine conflict information based on the infrared thermal image data and the audio data; determining responsible shop assistants based on the conflict information; and determining an alarm instruction based on the responsible shop assistant and the conflict information, and sending the alarm instruction to an alarm system.

Description

Technical Field

[0001] This specification relates to the field of intelligent identification, and in particular to an intelligent store conflict identification system and method. Background Art

[0002] In the operating environment of large stores (e.g., restaurants), due to the vast area of ​​the stores, it is difficult for managers to fully cover every corner of the stores during their inspections, especially during peak customer traffic periods when the number of customers surges and the mobility is extremely high, which further increases the difficulty of management. In this situation, sudden conflicts will have a serious impact on the normal operation of the store and the customer experience. However, due to the limited vision and reaction speed of managers, some conflicts are often not detected at the first time, and effective measures cannot be taken immediately to deal with them, which may lead to further deterioration of the situation.

[0003] Therefore, it is necessary to provide an intelligent store conflict identification system and method that can quickly identify and classify conflict events and instantly trigger an alarm mechanism, effectively improving management efficiency and customer service experience, and improving store safety. Summary of the invention

[0004] One or more embodiments of the present specification provide a smart store conflict identification system. The smart store conflict identification system includes: an infrared thermal imager, an audio acquisition device, an alarm system, a communication device, and a processor; the infrared thermal imager is set in a preset area of ​​the store, and the infrared thermal imager is configured to acquire infrared thermal imaging data of the preset area; the audio acquisition device is set in the preset area of ​​the store, and the audio acquisition device is configured to acquire audio data of the preset area; the alarm system is configured to send conflict information to the responsible clerk and execute an alarm based on an alarm instruction; the communication device is configured to realize data communication between the infrared thermal imager, the audio acquisition device, the alarm system, and the processor; the processor is configured to: determine the conflict information based on the infrared thermal imaging data and the audio data; determine the responsible clerk based on the conflict information; and determine the alarm instruction based on the responsible clerk and the conflict information, and send the alarm instruction to the alarm system.

[0005] One or more embodiments of the present specification provide a smart store conflict identification method. The method includes: obtaining infrared thermal imaging data of a preset area and audio data of the preset area; determining conflict information based on the infrared thermal imaging data and the audio data; determining a responsible clerk based on the conflict information; and determining an alarm instruction based on the responsible clerk and the conflict information, and sending the alarm instruction to an alarm system.

[0006] One or more embodiments of the present specification provide a computer-readable storage medium, wherein the storage medium stores computer instructions. When a computer reads the computer instructions in the storage medium, the computer executes a smart store conflict identification method. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] This specification will be further described in the form of exemplary embodiments, which will be described in detail by the accompanying drawings. These embodiments are not restrictive, and in these embodiments, the same number represents the same structure, wherein:

[0008] Figure 1 is a schematic diagram of an application scenario of a smart store conflict identification system according to some embodiments of this specification;

[0009] Figure 2 is an exemplary schematic diagram of a smart store conflict identification system according to some embodiments of this specification;

[0010] Figure 3 is an exemplary flow chart of a smart store conflict identification method according to some embodiments of this specification;

[0011] Figure 4 is an exemplary schematic diagram of a conflict prediction model according to some embodiments of this specification;

[0012] Figure 5 It is an exemplary schematic diagram of a conflict prediction model according to other embodiments of this specification. DETAILED DESCRIPTION

[0013] In order to more clearly illustrate the technical solutions of the embodiments of this specification, the following is a brief introduction to the drawings required for the description of the embodiments. Obviously, the drawings described below are only some examples or embodiments of this specification. For ordinary technicians in this field, this specification can also be applied to other similar scenarios based on these drawings without creative work. Unless it is obvious from the language environment or otherwise explained, the same reference numerals in the figures represent the same structure or operation.

[0014] It should be understood that the "system", "device", "unit" and / or "module" used herein are a method for distinguishing different components, elements, parts, portions or assemblies at different levels. However, if other words can achieve the same purpose, the words can be replaced by other expressions.

[0015] Unless the context clearly indicates an exception, the words "a", "an", "an" and / or "the" do not refer to the singular and may also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements.

[0016] Flowcharts are used in this specification to illustrate the operations performed by the system according to the embodiments of this specification. It should be understood that the preceding or following operations are not necessarily performed precisely in order. Instead, the steps may be processed in reverse order or simultaneously. At the same time, other operations may also be added to these processes, or one or more operations may be removed from these processes.

[0017] Figure 1 It is a schematic diagram of an application scenario of the intelligent store conflict identification system shown in some embodiments of this specification.

[0018] In some embodiments, Figure 1 As shown, the application scenario 100 of the intelligent store conflict identification system may include a store 110, a processor 120, a network 130, a storage device 140, and a user terminal 150, etc.

[0019] In some embodiments, the application scenarios 100 of the smart store conflict identification system may include multiple scenarios, such as application scenarios of large stores, etc. Large stores may include multiple scenarios, such as large catering stores, large hotels, etc. This specification takes a large catering store as an example for explanation.

[0020] In some application scenarios, the smart store conflict identification system 200 can be used to provide conflict information, conflict warnings, and reporting to management personnel. The smart store conflict identification system 200 can implement the methods and / or processes disclosed in this application to quickly identify and alarm conflict events. For more information about the smart store conflict identification system 200, please refer to Figure 2 Related description.

[0021] Store 110 refers to a large store monitored and identified by the intelligent store conflict identification system, such as a large restaurant store. In some embodiments, one or more infrared thermal imagers, audio acquisition devices, etc. may be installed at different locations in the store to acquire relevant data in a preset area within the store. For more information about infrared thermal imagers, audio acquisition devices, etc., please refer to Figure 2 and related content.

[0022] The processor 120 may process data and / or information obtained from other devices or system components. The processor may execute program instructions based on these data, information and / or processing results to perform one or more functions described in this application. For example, the processor 120 may determine conflict information based on infrared thermal imaging data and audio data. For another example, the processor 120 may determine the responsible clerk based on the conflict information; and determine an alarm instruction based on the responsible clerk and the conflict information, and send the alarm instruction to the alarm system.

[0023] In some embodiments, the processor 120 may include one or more sub-processing devices (e.g., a single-core processing device or a multi-core multi-core processing device). As an example only, the processor 120 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), an application-specific instruction processor (ASIP), a graphics processing unit (GPU), a physical processing unit (PPU), a digital signal processor (DSP), a microprocessor, etc., or any combination thereof.

[0024] The network 130 can connect the components of the smart store conflict identification system 200 and / or connect the system with external resources. The network 130 enables communication between the components and with other components outside the system to facilitate the exchange of data and / or information. For example, the processor 120 can obtain infrared thermal imaging data and audio data from the storage device 140 through the network 130.

[0025] In some embodiments, the network 130 may be any one or more of a wired network or a wireless network. For example, the network 130 may include a cable network, an optical fiber network, a telecommunications network, the Internet, a local area network (LAN), a wireless local area network (WLAN), a metropolitan area network (MAN), a bus within a device, a line within a device, etc., or any combination thereof. The network connection between the various parts may be in one of the above-mentioned ways, or in multiple ways. In some embodiments, the network may be a point-to-point, shared, centralized, or other topological structure, or a combination of multiple topological structures.

[0026] The storage device 140 may be used to store data and / or instructions. The storage device 140 may include one or more storage components, each of which may be an independent device or a part of another device. In some embodiments, the storage device 140 may include a random access memory (RAM), a read-only memory (ROM), a mass storage device, etc., or any combination thereof.

[0027] In some embodiments, when a conflict occurs between a customer and / or a store clerk in the store 110, the infrared thermal imager and the audio acquisition device can be used to timely acquire information about the conflict scene, and after being processed by the processor 120, the conflict characteristics of the store 110 are determined, and the conflict information is promptly sent to the responsible store clerk and an alarm operation is performed. When processing the conflict, the processor 120 can acquire data on the storage device 140 or save the data on the storage device 140.

[0028] The user terminal 150 refers to one or more terminal devices or software used by the user 160. In some embodiments, one or more users may use the user terminal 150. In some embodiments, the user terminal 150 may include a mobile phone 150-1, a tablet 150-2, and a computer 150-3.

[0029] The user 160 may include all employees of the store 110. For example, when an unexpected conflict occurs, the processor 120 determines the responsible employee based on the conflict information, and the responsible employee can obtain the conflict information in time through the user terminal 150 and handle the unexpected event more specifically based on the conflict information.

[0030] It should be noted that the application scenario 100 of the smart store conflict identification system is provided for illustrative purposes only and is not intended to limit the scope of this specification. For those of ordinary skill in the art, various modifications or changes can be made according to the description of this specification. For example, the application scenario 100 of the smart store conflict identification system may also include a cloud storage. For another example, the application scenario 100 of the smart store conflict identification system may be implemented on other devices to achieve similar or different functions. However, changes and modifications will not deviate from the scope of this specification.

[0031] Figure 2 is an exemplary schematic diagram of a smart store conflict identification system according to some embodiments of this specification. Figure 2 As shown, the smart store conflict identification system 200 may include an infrared thermal imager 210 , an audio acquisition device 220 , an alarm system 230 , a communication device 240 and a processor 120 .

[0032] The infrared thermal imager 210 refers to an instrument that uses infrared to perform non-contact testing, such as a refrigerated infrared thermal imager, an industrial temperature measurement infrared thermal imager, and the like.

[0033] In some embodiments, the infrared thermal imager 210 may be disposed at a first position in a preset area.

[0034] The preset area refers to an area that needs to be monitored and is preset in the store 110. For example, when the store is a large-scale restaurant, the large-scale restaurant may include at least one preset area. The lobby, at least one private room, the service counter, etc. may each be a preset area.

[0035] The first position refers to a position of the infrared thermal imager 210 that is pre-selected and set based on requirements within a preset area.

[0036] In some embodiments, a plurality of infrared thermal imagers 210 may be pre-installed in the store 110 .

[0037] In some embodiments, a preset area may be provided with at least one infrared thermal imager 210. The infrared thermal image data output by the infrared thermal imager 210 may cover the area of ​​the preset area.

[0038] In some embodiments, the number of infrared thermal imagers 210 is positively correlated with the number of preset areas. For example, the more preset areas there are in the store 110 , the more infrared thermal imagers 210 there are.

[0039] In some embodiments, the infrared thermal imager is configured to acquire infrared thermal imaging data.

[0040] The audio acquisition device 220 refers to a device for acquiring audio data of a preset area. For example, the audio acquisition device may be at least one of a microphone, a recording device, a wearable device, and the like.

[0041] In some embodiments, the audio acquisition device 220 may be disposed at a second position in a preset area.

[0042] The second position refers to a position of the audio acquisition device 220 that is pre-selected and set based on needs within a preset area.

[0043] In some embodiments, at least one audio acquisition device 220 may be set in a preset area to ensure that the audio acquisition device 220 can collect audio data in the preset area.

[0044] In some embodiments, the audio acquisition device 220 is configured to acquire audio data.

[0045] In some embodiments, one or more audio acquisition devices 220 may be provided in the store 110 .

[0046] In some embodiments, the number of audio acquisition devices 220 may be positively correlated with the number of preset areas. For example, the more preset areas there are in the store 110 , the more audio acquisition devices 220 there are.

[0047] The alarm system 230 is a system for sending conflict information to a store clerk in charge and issuing an alarm.

[0048] In some embodiments, the alarm system 230 is configured to execute an alarm based on an alarm instruction in response to sending conflicting information to a responsible store clerk.

[0049] In some embodiments, the processor may transmit the responsible clerk to the communication device, and the communication device receives the responsible clerk and transmits the responsible clerk to the alarm system 230. The alarm system 230 may send conflict information to the responsible clerk through the user terminal 150 and execute an alarm based on the alarm instruction.

[0050] The communication device 240 refers to a device used to transmit and exchange data information between different devices in the smart store conflict identification system 200 .

[0051] In some embodiments, the communication device 240 is configured to realize data communication between the infrared thermal imager 210, the audio acquisition device 220, the alarm system 230 and the processor 120. For example, when a sudden conflict event occurs, the infrared thermal image data output by the infrared thermal imager 210 and the audio data output by the audio acquisition device 220 are sent to the processor 120 through the communication device 240. For another example, the processor 120 determines the conflict information based on the infrared thermal image data and the audio data, and transmits the conflict information to the alarm system 230 through the communication device 240. For another example, the alarm system 230 sends the conflict information to the responsible clerk through the user terminal 150.

[0052] In some embodiments, the communication device 240 can realize data transmission and information exchange between different devices through a variety of communication technologies and interfaces, such as serial communication, network interface, bus interface, etc.

[0053] In some embodiments, the processor is configured to determine conflict information based on the infrared thermal imaging data and the audio data; determine the responsible clerk based on the conflict information; determine an alarm instruction based on the responsible clerk and the conflict information, and send the alarm instruction to the alarm system. For more information about the processor, see Figure 1 .

[0054] In some embodiments, the smart store conflict identification system 200 also includes an order monitoring device (not shown in the figure).

[0055] The order monitoring device refers to a device used to monitor the order information of the dining table, such as QR code scanning ordering software, user terminals for placing orders, etc.

[0056] In some embodiments, the order monitoring device is configured to monitor order information of a table.

[0057] In some embodiments, the smart store conflict identification system 200 also includes a network monitoring device and a vibration sensor (not shown in the figure).

[0058] Network monitoring devices refer to equipment used to monitor signal conditions in preset areas of a store, such as Wi-Fi probes, Bluetooth beacons, etc.

[0059] Signals refer to data signals transmitted in telecommunication networks. For example, signals may include but are not limited to Wi-Fi, mobile signal data, Bluetooth signals, etc.

[0060] In some embodiments, the processor 120 may determine the population density data of the preset area based on the network monitoring device. For example, the more network signals in the preset area monitored by the network monitoring device, the more people there are in the preset area. The processor may determine the population density data based on the number of people and the size of the preset area.

[0061] In some embodiments, the network monitoring device is set in a preset area of ​​the store, and the network monitoring device is configured to monitor and obtain the crowd density data of the preset area. In some embodiments, the network monitoring device is set at a third position in the preset area. The third position refers to the location of the network monitoring device pre-selected and set based on demand in the preset area. At least one network monitoring device is set in a preset area.

[0062] In some embodiments, one or more network monitoring devices may be installed in the store 110 .

[0063] In some embodiments, the number of network monitoring devices is positively correlated with the number of preset areas. For example, the more preset areas there are in the store 110, the more network monitoring devices there are.

[0064] The vibration sensor refers to a device used to monitor vibration data in a preset area of ​​a store, for example, at least one of a piezoelectric vibration sensor and an electric vibration sensor.

[0065] In some embodiments, the vibration sensor is set at a preset position in a preset area, and the vibration sensor is configured to monitor vibration data at the preset position. The preset position refers to a position of the vibration sensor that is pre-selected and set based on requirements in the preset area. At least one vibration sensor is set in a preset area.

[0066] In some embodiments, one or more vibration sensors may be provided in the store 110. In some embodiments, the number of vibration sensors is positively correlated with the number of preset areas. For example, the more preset areas in the store 110, the more vibration sensors there are.

[0067] In some embodiments, the processor 120 is configured to determine conflict information based on infrared thermal imaging data and audio data; determine the responsible clerk based on the conflict information; and determine an alarm instruction based on the responsible clerk and the conflict information, and send the alarm instruction to the alarm system.

[0068] In some embodiments, the processor is further configured to determine monitoring personnel information based on the infrared thermal imaging data and the order information; and determine conflict information based on the monitoring personnel information and the audio data.

[0069] In some embodiments, the processor is further configured to determine conflict information through a conflict prediction model based on monitored personnel information, audio data, vibration data, and personnel density data, and the conflict prediction model is a machine learning model.

[0070] In some embodiments, the processor is further configured to determine the conflict risk of a preset area based on the conflict information; and determine the responsible clerk based on the conflict risk.

[0071] In some embodiments, the processor is further configured to determine a risk change trend based on the conflict risk and the regional environment of a preset area; and determine a responsible clerk based on the risk change trend.

[0072] For more information on the above, see Figures 3 to 5 Related description.

[0073] Figure 3 FIG. 1 is an exemplary flow chart of a smart store conflict identification method according to some embodiments of this specification. Figure 3 As shown, the process 300 includes the following steps 310 to 340. In some embodiments, the process 300 may be executed by the processor 120.

[0074] In some embodiments, the processor 120 can obtain infrared thermal imaging data of a preset area and audio data of a preset area; determine conflict information based on the infrared thermal imaging data and the audio data; determine the responsible clerk based on the conflict information; and determine an alarm instruction based on the responsible clerk and the conflict information, and send the alarm instruction to the alarm system.

[0075] Step 310: Acquire infrared thermal imaging data of a preset area and audio data of a preset area.

[0076] For more information about preset areas, see Figure 2 Related description.

[0077] Infrared thermal imaging data refers to relevant data about a preset area obtained through infrared thermal imaging technology. Infrared thermal imaging technology refers to the technology of obtaining temperature distribution information on the surface of a biological body using the infrared spectrum radiated by an object. Infrared thermal imaging data may include at least one of infrared images and temperature data.

[0078] The infrared thermal imaging data in some embodiments of the present specification may be replaced by point cloud data or other data that can identify or track organisms, which is not limited here.

[0079] In some embodiments, the processor may obtain infrared thermal imaging data of at least one preset area in real time or at a scheduled time through infrared thermal imagers disposed in preset areas of the store.

[0080] Audio data refers to data representing sound information in a preset area. For example, the audio data may include data of at least one segment of recorded audio about at least one preset area.

[0081] In some embodiments, the processor may obtain audio data of at least one preset area in real time or at a scheduled time through audio acquisition devices disposed in preset areas of the store.

[0082] For more information about infrared thermal imaging cameras and audio acquisition equipment, see Figure 2 Related description.

[0083] Step 320: determining conflict information based on the infrared thermal imaging data and the audio data.

[0084] Conflict information refers to information related to conflicts that may arise in the offline management of a store.

[0085] In some embodiments, the conflict information includes conflicting personnel relationships. The conflicting personnel relationships refer to the relationships between conflicting personnel. The conflicting personnel relationships may include at least one of conflicts between store clerks and customers, conflicts between customers, and the like.

[0086] In some embodiments, the store may be a large catering store. A large catering store refers to a store whose store size exceeds a size threshold. In some embodiments, the store size may be determined based on one or more of the store's seat capacity, store floor area, number of employees, etc. Correspondingly, the size threshold may include one or more of a seat capacity threshold, a store floor area threshold, number of employees threshold, etc. The size threshold may be pre-set by a person skilled in the art. For example, the size threshold may be a store floor area threshold of 500 square meters; if the store floor area exceeds 500 square meters, it is a large catering store.

[0087] In some embodiments, the processor may determine conflict information in a variety of ways based on infrared thermal imaging data and audio data. For example, the processor may identify infrared thermal imaging data of at least one preset area and determine the number of people in the infrared thermal imaging data. The processor may perform human voice separation and recognition processing on audio data of at least one preset area to determine human voice track information; based on the number of people and the human voice track information, determine whether there is a conflict, and determine conflict information in response to the existence of a conflict. The processor may determine conflict information in the following ways.

[0088] The method for identifying the infrared thermal imaging data of at least one preset area may be an image recognition technology or other recognition-capable technology, which is not limited here.

[0089] The vocal track information refers to the sound information related to the human voice. The vocal track information may include the vocal content and the vocal volume. The vocal content refers to the vocal content of the person identified in the vocal track information. The vocal volume refers to the volume of the vocal track information.

[0090] The method for performing human voice separation and recognition processing on the audio data of at least one preset area may include blind source separation, spectrum separation, speech recognition technology, etc., or may be other human voice separation and recognition technologies, which are not limited here.

[0091] In some embodiments, the processor may determine whether a conflict exists and determine conflict information in a variety of ways based on the number of people and vocal track information.

[0092] In some embodiments, the conflict information includes a conflict area. The conflict area refers to a preset area where a conflict may occur.

[0093] In some embodiments, when the number of people in a preset area is greater than a threshold number, and the human voice content corresponding to the preset area contains non-standard words, the processor can determine that there is a conflict in the preset area and determine the preset area as the conflict area in the conflict information.

[0094] The number threshold refers to the number of people that must be met when conflicting information may exist in a preset area. For example, the number threshold is 2 people.

[0095] Non-standard words refer to words that may cause conflicts, for example, non-standard words are commonly used quarrel words, swear words, etc. Non-standard words can be preset by technical personnel.

[0096] In some embodiments, the processor may determine whether there are non-standard words in the human voice content in a variety of ways. For example, the processor may search the human voice content based on the non-standard words, and when the non-standard words are found, it is considered that there are non-standard words in the human voice content.

[0097] In some embodiments, when the number of people is greater than a number threshold and the volume of human voices is greater than a volume threshold, the processor may determine that there is a conflict in the preset area and determine the preset area as a conflict area.

[0098] The volume threshold refers to the threshold of the volume at which a conflict may occur. The volume threshold can be preset by a technician. In some embodiments, the volume threshold is the average volume of human voices in historical conflict information. Historical conflict information includes conflict information that has been processed and in which a conflict has occurred.

[0099] In some embodiments, when the human voice content corresponding to the conflict area contains preset keywords, the processor can determine that there is a conflict in the conflict area, and determine the conflicting personnel relationship corresponding to the conflict area as a conflict between a clerk and a customer.

[0100] The preset keywords refer to pre-set keywords, such as waiter, clerk, waiter, etc. When the human voice content contains the above preset keywords, it indicates that the human voice content is a communication between a customer and a clerk, so that the conflicting personnel relationship corresponding to the conflict area can be judged to be a conflict between a clerk and a customer by the preset keywords contained in the human voice content. In response to the human voice content not containing the preset keywords, it is judged that the conflicting personnel relationship corresponding to the conflict area is a conflict between customers.

[0101] In some embodiments, the processor may determine monitoring personnel information based on infrared thermal imaging data and order information; and determine conflict information based on monitoring personnel information and audio data.

[0102] Order information refers to various data related to a customer's order. For example, order information may include at least one of the table number, order time, and dish information of at least one table. In some embodiments, the processor may obtain order information through an order monitoring device. For more information about the order monitoring device, see Figure 2 Related description.

[0103] In some embodiments, the order information includes order clerk information. The order clerk information refers to the relevant information of the clerk responsible for the order. After the customer places an order through the QR code of the table, the processor can assign a clerk to each order according to a preset assignment rule; and write the assigned clerk into the order clerk information of the order information.

[0104] In some embodiments, the preset allocation rules may be multiple. For example, the preset allocation rules may include that different preset areas have different responsible clerks. The processor may determine the area to which the table belongs based on the table number of the customer's order, and determine the order clerk information of the table. For another example, the preset allocation rules may include that when ordering manually, different clerks' user terminals correspond to different responsible clerks; at this time, the clerks each place an order for the customer through their own user terminals, and the processor may determine the order clerk information based on the clerk's user terminal.

[0105] The monitoring personnel information refers to the information about the person suspected of having a conflict. The person may include at least one of a customer, a clerk, etc. For example, the monitoring personnel information may include at least one of the person suspected of having a conflict, the area where the person is located, and the order information of the table associated with the person.

[0106] When the number of tables associated with the person suspected of having a conflict exceeds one (for example, the customer has placed orders for tables 2, 3, and tables corresponding to other customers with whom the customer has had a conflict, etc.), the order information of the tables associated with the person suspected of having a conflict also includes the order information of other tables.

[0107] In some embodiments, the processor can identify the infrared thermal imaging data of each preset area, determine the position of each person in the infrared image, and construct a topological map by taking the position of each person as a node. The processor can determine the number of nodes and whether there is a central node based on the topological map. If there is a central node, and the number of central nodes is greater than the central node number threshold, then the information related to the personnel corresponding to all central nodes is the monitoring personnel information; if there is no central node, then there is no monitoring personnel information in the preset area.

[0108] The central node number threshold can be preset by a technician. For example, the central node number threshold can be 2.

[0109] In some embodiments, the edges of the topological graph have no direction. The number of nodes may correspond to the number of people in the preset area.

[0110] In some embodiments, the processor may determine whether there is a central node in a variety of ways. For example, the processor may determine a node that satisfies a central node condition as a central node. For example, the central node condition may be: a node that all nodes can reach via 0 to a preset number of nodes is a central node (i.e., other nodes in the topology graph can reach the central node through a transmission step that does not exceed the preset number of nodes), and / or a node located in the central range of the topology graph is a central node (i.e., the central node is not at the edge of the topology graph), etc.

[0111] The preset number of nodes can be preset by the technicians. For example, the preset number of nodes is 3; at this time, all other nodes can reach the central node through the transmission of no more than 3 nodes.

[0112] In some embodiments, the central node condition also includes determining that the number of central nodes must be two or more; two or more central nodes may indicate that there are two parties suspected of conflict.

[0113] In some embodiments, after confirming the central node, the processor can identify the dining table where the person corresponding to the central node is located and retrieve the order information of the table or multiple tables (such as when the people corresponding to the central node are from different tables), and determine the monitoring personnel information.

[0114] In some embodiments of the present specification, by evaluating the distribution of personnel in each preset area and constructing a topological map, it is possible to determine whether there are persons suspected of being involved in a conflict, and thus process the relevant data of the persons suspected of being involved in a conflict, which can reduce the amount of data processing and improve the accuracy of determining the conflict information.

[0115] In some embodiments, the processor can determine the conflict information in a variety of ways based on the monitoring personnel information and audio data. For example, the processor can obtain the preset area where the person in the monitoring personnel information is located, retrieve the audio data of the preset area, and determine whether the human voice content in the audio data contains preset keywords. If it contains preset keywords, it is determined that there is a conflict, and the conflicting personnel relationship in the preset area is determined as a conflict between a clerk and a customer. For more information about non-standard words, volume thresholds, and preset keywords, please refer to Figure 2 The relevant description above.

[0116] In some embodiments, the processor can determine the target audio data of the order information based on the monitoring personnel information and the audio data; based on the target audio data, perform vocal separation to identify whether there are irregular words in the vocal track information; in response to the presence of irregular words and the vocal volume of the vocal track information is greater than a volume threshold, it is determined that there is a conflict, and a preset area is determined as the conflict area, and the volume threshold is related to the order information; in response to the recognition of preset keywords in the vocal track information, it is determined that there is a conflict, and the conflict information is determined as a conflict between a clerk and a customer, otherwise, the conflict information is determined as a conflict between a customer and a customer.

[0117] The target audio data refers to audio data associated with the monitoring person information.

[0118] In some embodiments, the processor can determine the target audio data of the order information based on the monitoring personnel information and the audio data in a variety of ways. For example, the order information associated with the person suspected of conflict in the monitoring personnel information is obtained, the table number of the order information is extracted, and the audio data collected near the geographical location of the table corresponding to the table number is used as the target audio data. For another example, the processor can use the audio data collected by the wearable device of the staff responsible for the order in the order clerk information as the target audio data.

[0119] The wearable device may be a type of audio acquisition device, for example, a wearable device may include an intercom with a recording function, a hanging headset, etc.

[0120] In some embodiments, the processor performs vocal separation and identifies whether there are irregular words in the vocal track information based on the target audio data, which may include: performing vocal separation and identification processing on the target audio data to determine the vocal track information; and determining whether there are irregular words based on the vocal content of the vocal track information. The method of performing vocal separation and identification processing on the target audio data to determine the vocal track information is similar to the method of performing vocal separation and identification processing on the audio data to determine the vocal track information, the difference being that the former performs vocal separation and identification processing on the target audio data, while the latter performs vocal separation and identification processing on the audio data.

[0121] In some embodiments, in response to identifying a preset keyword in the vocal track information, the processor may determine that the conflict information is a conflict between a clerk and a customer, and vice versa, the conflict information is a conflict between customers. For more information about vocal separation processing, vocal track information, non-standard words, vocal content based on vocal track information, methods for determining whether there are non-standard words, preset keywords, and methods for identifying preset keywords in vocal track information, see Figure 3 The above description.

[0122] In some embodiments, the processor may determine the preset area as a conflict area in response to the presence of non-standard words and the human voice volume of the human voice track information being greater than a volume threshold, where the volume threshold is related to the order information. For more information about the volume threshold and order information, see Figure 3 The above description.

[0123] In some embodiments, the volume threshold may be positively correlated to the difference between the order time and the current time of the order information. When the difference between the order time and the current time is smaller, that is, the order time is closer to the current time, the volume threshold is smaller; when the difference between the order time and the current time is larger, the volume threshold is larger.

[0124] In some embodiments, the processor can determine the volume threshold through a first preset relationship table. The first preset relationship table refers to a preset table including the correspondence between the time difference and the volume threshold. The time difference is the difference between the order time of the order information and the current time. In some embodiments, the first preset relationship table can be constructed by the processor based on historical data or by a technician based on historical experience.

[0125] The smaller the difference between the order time and the current time, the more likely it is that the customer at the table has just placed an order. At this time, the customer may have many needs, such as waiting for the meal, inquiring about the dishes, etc. At this time, the probability of a conflict between the customers at the table may be greater than that of the customers who have already eaten. Therefore, the volume threshold is related to the difference between the order time and the current time of the order information, so as to set a more reasonable volume threshold, which is conducive to more timely detection of conflict information.

[0126] In some embodiments, the processor may identify a preset keyword in the vocal track information, and the conflict information is a conflict between a clerk and a customer, and vice versa, the conflict information is a conflict between a customer and a customer. For more information about preset keywords and conflict information, see Figure 3 Related description.

[0127] In some embodiments of the present specification, by first determining the monitoring personnel information and then retrieving the audio data related to the monitoring personnel information, focusing on processing specific audio data can reduce the amount of data processing and speed up processing.

[0128] Step 330, determining the responsible clerk based on the conflict information.

[0129] The responsible clerk refers to the person responsible for handling a certain period of time or a preset area or a certain type of accident. For example, the responsible clerk is the regional manager of the preset area, the on-duty person, the on-duty person's superior, etc.

[0130] In some embodiments, the processor can determine the responsible clerk in a variety of ways based on the conflict information. For example, the processor can obtain the conflict area of ​​the conflict information and the duty information of the conflict area; the processor determines the responsible clerk based on the duty information of the conflict area. The duty information is pre-set. For example, the processor can determine the responsible clerk based on the conflict personnel relationship of the conflict information; for example, the conflict between a customer and a clerk, and the conflict between customers, may have different responsible clerks.

[0131] In some embodiments, the processor may determine the conflict risk of a preset area based on the conflict information; and determine the responsible clerk based on the conflict risk.

[0132] Conflict risk refers to the security risk that conflict information may cause. Conflict risk can be expressed by the value of the danger coefficient. The higher the value, the greater the risk. For example, when the conflicting personnel relationship in the conflict information is a conflict between customers, the conflict risk is high (the conflict between customers may be more difficult to handle); when it is a conflict between a customer and a clerk, the conflict risk is low.

[0133] In some embodiments, the processor determines the conflict risk of the preset area based on the conflict information, which may include: adjusting a default risk factor based on the conflicting personnel relationship in the conflict information, and determining the adjusted default risk factor as the conflict risk of the preset area. The default risk factor is pre-set based on the preset area.

[0134] The default risk factor refers to the risk factor pre-set for a preset area. The default risk factors of different preset areas may be the same or different. In some embodiments, the default risk factor of each preset area may be preset in advance based on the number of dangerous goods in the preset area; the more dangerous goods there are, the higher the default risk factor.

[0135] Dangerous goods may refer to items that may cause danger, such as knives and forks, hot pots, etc.

[0136] In some embodiments, if the conflicting personnel relationship is a conflict between a customer and a store clerk, the processor may reduce the default risk factor and use the reduced default risk factor as the conflict risk for the preset area; if the conflicting personnel relationship is a conflict between customers, the processor may increase the default risk factor and use the increased default risk factor as the conflict risk for the preset area.

[0137] In some embodiments of the present specification, conflicts between customers may be easily driven by emotions, so the conflict risk is set to a larger value so that the conflict can be handled promptly and adequately.

[0138] In some embodiments, the processor may determine the responsible clerk based on the magnitude of the conflict risk. For example, when the conflict risk of the preset area is high, the responsible clerk may be the superior of the on-duty personnel. The greater the conflict risk, the higher the level and qualifications of the superior. When the conflict risk of the preset area is low, the responsible clerk may be the on-duty personnel.

[0139] In some embodiments of the present specification, by considering the conflict risk that may be caused by the conflict in the preset area and determining the responsible clerk according to the magnitude of the conflict risk, high-risk conflicts can be paid attention to by higher-level managers, so that the conflicts can be better resolved or handled.

[0140] In some embodiments, the processor may determine a risk change trend based on the conflict risk and the regional environment of a preset area; and determine a responsible clerk based on the risk change trend.

[0141] The regional environment refers to the environmental characteristics within the preset area. For example, the regional environment may include the area size, area type, number of devices (such as the number of infrared thermal imagers 210, audio acquisition devices 220, etc.) of the preset area. Among them, the area type refers to the type of area divided according to the specific function or purpose of the area, such as lobby, private room, toilet, kitchen, cashier counter, corridor, etc.

[0142] In some embodiments, the processor may determine the regional environment of the preset area in a variety of ways. For example, the regional environment may be preset by the clerk through the user terminal 150 and stored in the storage device 140, and the processor 120 may obtain the regional environment through the storage device 140 to determine the regional environment of the preset area.

[0143] In some embodiments, the regional environment can be characterized by the complexity of the regional environment. The complexity of the regional environment refers to the concentration of devices in the regional environment. The complexity of the regional environment can be represented by a numerical value, and the larger the numerical value, the more complex the regional environment. The devices in the regional environment include but are not limited to infrared thermal imagers 210, audio acquisition devices 220, etc. The processor can determine the complexity of the regional environment in a variety of ways based on the number of devices in the regional environment and the location distribution of the devices. For example, if the number of devices in the regional environment is large and the location distribution is uniform or concentrated, the higher the value of the complexity of the regional environment; if the number of devices in the regional environment is small and the distribution is dispersed, the lower the value of the complexity of the regional environment.

[0144] The risk change trend refers to data that evaluates the change in the conflict risk of a preset area in the future. In some embodiments, the risk change trend can be represented by a numerical value (e.g., a numerical value between 0 and 1). The value of the risk change trend is greater than 0.5 and the larger the value is, the more serious the conflict risk will be in the future. The value of the risk change trend is less than or equal to 0.5 and the smaller the value is, the less serious the conflict risk will be in the future. The risk change trend can also be represented by any other feasible method, for example, by text, level, etc.

[0145] In some embodiments, the processor can determine the risk change trend in a variety of ways based on the conflict risk and the complexity of the regional environment. For example, the risk change trend is positively correlated with the conflict risk and the complexity of the regional environment. The higher the complexity of the regional environment of the preset area and the higher the conflict risk, the larger the value of the risk change trend, and the risk change trend is a gradually increasing trend. If the complexity of the regional environment of the preset area is lower and the conflict risk is lower, the value of the risk change trend is smaller, and the risk change trend is a gradually reducing trend.

[0146] In some embodiments, the processor can determine the risk change trend based on a preset formula. Exemplarily, the risk change trend can be calculated based on the following formula: Risk change trend = W1* conflict risk + W2* complexity of the regional environment. W1 and W2 are the weights of the conflict risk and the complexity of the regional environment, respectively, and can be preset. If the risk change trend value is greater than 0.5, the risk change trend is a gradually increasing trend. The larger the value, the more serious the conflict risk. If the risk change trend value is less than or equal to 0.5, the risk change trend is a gradually reducing trend. The smaller the value, the greater the degree of reduction of the conflict risk.

[0147] In some embodiments, the processor can determine the responsible clerk in a variety of ways based on the risk change trend. For example, if the risk change trend is a gradually decreasing trend, the responsible clerk is the duty officer of the preset area. If the risk change trend is a gradually increasing trend, the responsible clerk is the duty officer of the preset area and the superior of the duty officer, etc. In some embodiments, the duty officer can be determined by a duty table.

[0148] In some embodiments, determining the responsible clerk based on the risk change trend further includes: determining the responsible clerk based on the risk change trend and order information. For more information about order information, see Figure 3 Step 320 and related instructions.

[0149] In some embodiments, the processor can determine the responsible clerk in a variety of ways based on the risk change trend and order information. For example, if the risk change trend is a gradually easing trend, the responsible clerk is the on-duty staff in the preset area and the clerk responsible for the order of the table; if the risk change trend is a gradually increasing trend, the responsible clerk is the clerk in the preset area responsible for the order of the table, the on-duty staff, and the superior of the on-duty staff.

[0150] In some embodiments of the present specification, the processor can determine the clerk responsible for the order of the table through the order information. The clerk often has a deeper understanding of the conflicting parties and can therefore provide more specific and targeted help to resolve the conflict.

[0151] In some embodiments of the present specification, the processor may evaluate the risk change trend of the conflict risk in the future period. If the risk change trend shows a gradual easing trend, there is no need to further report it to a higher level of management to save human resources. On the contrary, if the risk change trend shows a gradual aggravation trend, it should be promptly reported to a higher level of management to ensure that the conflict can be responded to and resolved quickly and effectively.

[0152] Step 340, based on the responsible clerk and the conflict information, an alarm instruction is determined and sent to the alarm system.

[0153] The alarm instruction refers to the instruction used by the alarm system to execute the alarm. For example, the alarm instruction may include the responsible clerk, conflict information, and whether to send an alarm to multiple people. Whether to send an alarm to multiple people refers to whether to send conflict information to multiple clerks and multiple managers at the same time.

[0154] In some embodiments, the processor may determine the alarm instruction through a second preset relationship table. The second preset relationship table refers to a preset table including the corresponding relationship between the responsible clerk and the conflict information and the alarm instruction. In some embodiments, the second preset relationship table may be constructed by the processor based on historical data or by the technician based on historical experience.

[0155] In some embodiments, the alarm system may execute an alarm based on an alarm instruction; the alarm system may transmit the alarm instruction (such as conflict information) to a user terminal (such as a user terminal of a responsible clerk) via a communication device. In some embodiments, when the alarm instruction includes sending an alarm to multiple people, the alarm system may simultaneously transmit the conflict information to user terminals of multiple clerks or multiple managers via a communication device.

[0156] In large-scale catering stores, conflicts occasionally occur, but management personnel cannot be informed of them immediately and thus cannot handle them in time. In some embodiments of this specification, through infrared thermal imaging data and audio data of preset areas, conflict information can be determined in time, which is conducive to handling conflicts in time, improving the management efficiency and customer service experience of large-scale catering stores, and improving store safety.

[0157] It should be noted that the above description of process 300 is only for example and illustration, and does not limit the scope of application of this specification. For those skilled in the art, various modifications and changes can be made to process 300 under the guidance of this specification. However, these modifications and changes are still within the scope of this specification.

[0158] Figure 4 is an exemplary schematic diagram of a conflict prediction model according to some embodiments of the present specification.

[0159] In some embodiments, the processor can determine conflict information 460 through a conflict prediction model 450 based on the monitoring personnel information 410, audio data 420, vibration data 430 and personnel density data 440, and the conflict prediction model is a machine learning model. For more information about monitoring personnel information, audio data, and conflict information, please refer to Figure 3 And related instructions.

[0160] The vibration data 430 refers to data related to vibrations generated by the collision. For example, the vibration data 430 may include a vibration amplitude sequence or other vibration information. Other vibration information includes a sequence of vibration frequency components. The sequence of vibration frequency components refers to data collected by each vibration sensor within a period of time, that is, vibration data 430 at multiple time points within a period of time.

[0161] In some embodiments, the vibration sensor monitors the vibration at a preset position in a preset area and transmits the vibration data 430 to the processor via the communication device 240. The processor can obtain the vibration data 430 of the preset position in the preset area. For more information about the preset position, see Figure 2 Related description.

[0162] The population density data 440 refers to comprehensive information characterizing the number and distribution of people in a preset area. The population density data 440 can be represented by the number of people in a unit area, etc. The larger the number of people in a unit area, the denser the population in the corresponding preset area.

[0163] In some embodiments, the processor may obtain the densely populated data 440 in a variety of ways. For example, the processor may obtain network signals (such as Wi-Fi probes, Bluetooth beacons, mobile signal data, Wi-Fi hotspot data, etc.) through a network monitoring device, and then obtain the densely populated data 440.

[0164] In some embodiments, the processor may confirm the number of people based on the number of network signals, and determine the population density data 440 based on the number of people and the size of the preset area. For example, the processor may determine the ratio of the number of people to the area of ​​the preset area as the population density data 440.

[0165] The conflict prediction model 450 refers to a model for determining the conflict information 460 in the preset area. In some embodiments, the conflict prediction model can be a multimodal machine learning model. For example, the conflict prediction model is one or any combination of convolutional neural networks (CNN), recurrent neural networks (RNN), deep neural networks, long short-term memory networks, etc.

[0166] In some embodiments, the input of the conflict prediction model may be monitoring personnel information 410 , audio data 420 , vibration data 430 , and personnel density data 440 of a preset area, and the output may be conflict information 460 of the preset area.

[0167] In some embodiments, the conflict prediction model may include a conventional modality module, a low-noise modality module, a modality integration layer, and an output layer. For more information about the conflict prediction model, see Figure 5 And related instructions.

[0168] In some embodiments, the conflict prediction model can be obtained based on a large number of first training samples with first labels. The processor can input multiple first training samples with first labels into the initial conflict prediction model, construct a loss function through the first label and the result of the initial conflict prediction model, and iteratively update the initial conflict prediction model based on the loss function, wherein one round of iterative training includes: inputting a group of first training samples with first labels into the initial conflict prediction model, determining the loss function value through the first label and the output result of the initial conflict prediction model, and iteratively updating the parameters of the initial conflict prediction model based on the loss function value. When multiple rounds of iterations are performed until the iteration conditions are met, the training of the initial conflict prediction model is terminated and a trained conflict prediction model is obtained. Among them, the iteration method may include but is not limited to the gradient descent method, etc., and the iteration conditions may include one or more of the convergence of the loss function, the number of iterations reaching a preset number threshold, and the loss function value being less than a preset threshold.

[0169] In some embodiments, the first training sample may be sample monitoring personnel information, sample audio data, sample vibration data, and sample personnel density data corresponding to a sample preset area in the historical data.

[0170] In some embodiments, the first label may be actual conflict information (such as a conflict between a clerk and a customer or a conflict between customers) corresponding to a preset sample area of ​​the first training sample.

[0171] In some embodiments, the first training sample can be obtained through the storage device 140. In some embodiments, the first label of the conflict prediction model can be the actual conflict information subsequently corresponding to the sample preset area of ​​the first training sample, which can be determined based on manual records of the store clerk.

[0172] In some embodiments of the present specification, by real-time monitoring and acquisition of population density data of a preset area by a network monitoring device, combined with audio data and vibration data, the processor can more comprehensively grasp the on-site situation.

[0173] Figure 5 It is an exemplary schematic diagram of a conflict prediction model according to other embodiments of this specification.

[0174] In some embodiments, the conflict prediction model 450 may include a normal modality module 451 , a low-noise modality module 456 , a modality integration layer 453 , and an output layer 455 .

[0175] Conventional model module 451 refers to a model layer structure for determining visual modality information, auditory modality information, and tactile modality information. In some embodiments, conventional model module 451 includes visual modality layer 451-1, auditory modality layer 451-3, and tactile modality layer 451-2.

[0176] Modality refers to some ways of expressing or perceiving things. For example, each source or form of information can be called a modality. Multimodality refers to expressing or perceiving things from multiple modalities. For example, the processor can obtain different types of data in the same scene (for example, infrared thermal imaging data, audio data, etc.), or data obtained under different circumstances (such as different preset positions, different sensor angles). By analyzing the different types of data or data obtained in different circumstances, the processor can use the complementarity and redundancy of multiple modalities to analyze and summarize multimodal data, thereby improving the accuracy and comprehensiveness of the analysis.

[0177] In some embodiments, the visual modality layer 451 - 1 determines the visual modality information 452 - 1 based on the infrared thermal imaging data 510 - 2 .

[0178] The visual modality layer 451-1 refers to a model layer structure for determining visual modality information. In some embodiments, the visual modality layer 451-1 may be a machine learning model, such as a convolutional neural network model.

[0179] The input of the visual modality layer 451-1 may include infrared thermal imaging data 510-2 of at least one preset area, and the output may include visual modality information 452-1. For more information about infrared thermal imaging data, see Figure 3 Related description.

[0180] Visual modality information refers to information similar to information recognized by human vision. In some embodiments, the visual modality information includes contour features and position features of at least one organism. The organism may include animals, humans, etc. In some embodiments, the organism mainly refers to humans, such as store clerks and customers.

[0181] Contour features refer to features related to the outer contour of a target (i.e., an organism) in infrared thermal imaging data. For example, contour features can characterize the square grids occupied by the contour of the organism and the size of the contour formed by the square grids.

[0182] Position features refer to features related to the position coordinates of the target (i.e., the organism) in the infrared thermal imaging data. For example, position features can represent the position of a clerk in a store or a customer in a restaurant.

[0183] In some embodiments, the processor may train the visual modality layer in a variety of ways. For example, the second training sample for the visual modality layer may include sample infrared thermal imaging data. The second label corresponding to the second training sample may be the sample visual modality information corresponding to the group of second training samples. The sample infrared thermal imaging data may be infrared thermal imaging data of a store in historical data. In some embodiments, the second label may be obtained by manual annotation. For example, a technician may manually identify the infrared thermal imaging data, and mark the outline of the identified clerk or customer as an outline feature; and mark the position coordinates corresponding to the outline feature as a position feature.

[0184] The training process of the visual modality layer is similar to that of the conflict prediction model. For more information about the training process of the visual modality layer, see Figure 4 The above is a description of the conflict prediction model training.

[0185] In some embodiments of the present specification, by manually marking the second label, the visual modal layer can simulate the process of human eye recognition of images, so that high-level features recognized by the human eye can be extracted from the infrared thermal imaging data. High-level features refer to features that are closer to human understanding or perception. For example, the human eye can also recognize the contour, color, and position of an organism based on a small amount of data (far less than infrared thermal imaging data). At this time, the contour features and position features are high-level features. The visual modal layer can simulate the extraction process of human eye recognition (such as identifying contour features and position features, etc.) through a training process based on the second training sample and the second label.

[0186] In some embodiments, the auditory modality layer 451-3 determines the auditory modality information 452-3 based on the audio data 420. For more information about audio data, see Figure 3 Related description.

[0187] The auditory modality layer 451-3 refers to a model layer structure used to determine the auditory modality information 452-3. In some embodiments, the auditory modality layer 451-3 may be a machine learning model, such as a deep neural network model.

[0188] The input of the auditory modality layer 451 - 3 may include audio data 420 of at least one preset area, and the output may include auditory modality information 452 - 3 .

[0189] Auditory modal information refers to information that is similar to the information recognized by human hearing.

[0190] In some embodiments, the auditory modality information may include at least one of the voice age, voice gender, timbre, and speaking speed of the organism.

[0191] The voice age and voice gender may refer to the age and gender of a person in the audio data. The timbre may be described by a reference tone (reference frequency) and overtones (local frequencies). For example, a sound produced by a frequency vibration of 440 Hz may be called a reference tone; a sound with a frequency much higher than the reference tone may be called an overtone, such as a sound with a frequency that is two, three, four, and five times the reference frequency of 440 Hz, which is a overtone.

[0192] In some embodiments, the processor can train the auditory modality layer in a variety of ways. For example, the third training sample for the auditory modality layer can include sample audio data. The third label corresponding to the third training sample can be sample auditory modality information corresponding to the group of third training samples. The sample audio data can be audio data of a store in historical data. In some embodiments, the third label can be obtained by manual annotation. The manual annotation method and the annotation method of the visual modality layer can be referred to in Figure 5 Description of the annotations of the visual modality layer.

[0193] The training process of the auditory modal layer is similar to that of the conflict prediction model. For more information about the training process of the auditory modal layer, see Figure 4 The above is a description of the conflict prediction model training.

[0194] Similar to the above, in some embodiments of the present specification, by manually marking the third label, the auditory modal layer can simulate the process of human ear recognizing sound, thereby extracting the sound age, etc. similar to the human eye recognition result from the audio data.

[0195] In some embodiments, the tactile modality layer determines tactile modality information based on the vibration data.

[0196] The tactile modality layer 451-2 refers to a model layer structure used to determine the tactile modality information 452-2. In some embodiments, the tactile modality layer 451-2 may be a machine learning model, such as a deep neural network model.

[0197] The input of the tactile modality layer 451 - 2 may include vibration data 430 of at least one preset area, and the output may include tactile modality information 452 - 2 .

[0198] The tactile modality information 452 - 2 refers to information similar to information recognized by a person's tactile sense.

[0199] In some embodiments, the tactile modality information 452 - 2 includes the vibration type and vibration source of the non-biological object.

[0200] Vibration types may include frequency types and vibration cause types. Frequency types may include low frequency, medium frequency, high frequency, etc. For example, a frequency less than 10Hz is a low frequency; a frequency between 10Hz and 1000Hz is a medium frequency; a frequency greater than 1000Hz is a high frequency. Vibration cause types may include free vibration, forced vibration, self-excited vibration, etc. Free vibration refers to the periodic vibration of an object due to disturbance without the action of external force. Forced vibration refers to the vibration of an object caused by an external force. Self-excited vibration refers to the vibration of a system that continuously increases the amplitude of vibration through a positive feedback mechanism under certain conditions.

[0201] The vibration source refers to whether the vibration comes from a biological object or a non-biological object. For example, when a fan is running near the preset area, the vibration source may be a non-biological object.

[0202] In some embodiments, the processor can train the tactile modality layer in a variety of ways. For example, the fourth training sample for the tactile modality layer can include sample vibration data. The fourth label corresponding to the fourth training sample can be the sample tactile modality information corresponding to the group of training samples. The sample vibration data can be the vibration data of the store in the historical data. In some embodiments, the fourth label can be obtained by manual annotation. The manual annotation method and the annotation method of the visual modality layer can be referred to in Figure 5 Description of the annotations of the visual modality layer.

[0203] The training process of the tactile modality layer is similar to that of the conflict prediction model. For more information about the training process of the tactile modality layer, see Figure 4 The above is a description of the conflict prediction model training.

[0204] Similar to the above, in some embodiments of this specification, by manually marking the fourth label, the tactile modal layer can simulate the process of human tactile recognition of vibration, thereby extracting vibration types and vibration sources similar to human tactile recognition results from the vibration data.

[0205] In some embodiments, the modality integration layer 453 processes the visual modality information 452-1, the auditory modality information 452-3, the tactile modality information 452-2, the monitoring personnel information 410, and the order information 510-1 to obtain the first embedded feature 454. For more information about the monitoring personnel information and the order information, see Figure 3 Related description.

[0206] The modality integration layer 453 refers to a model layer structure for integrating multiple modality data. In some embodiments, the modality integration layer can be a machine learning model, for example, a deep neural network model, a long short-term memory network model, or any combination thereof.

[0207] The input of the modal integration layer 453 may include visual modal information 452-1, auditory modal information 452-3, tactile modal information 452-2, monitoring personnel information 410 of at least one preset area, and order information 510-1 of at least one preset area, and the output may include a first embedded feature 454.

[0208] The first embedded feature 454 may represent information input by the modality integration layer.

[0209] In some embodiments, the low noise modality module 456 is an embedding feature extraction layer 456-1, and the embedding feature extraction layer 456-1 determines a second embedding feature 457 based on the crowd density data 440. For more information about crowd density data, see Figure 4 Related description.

[0210] The low-noise modality module 456 refers to a model layer structure for determining the second embedding feature 457. In some embodiments, the low-noise modality module 456 may be a machine learning model, such as a neural network model.

[0211] The input of the low-noise modality module 456 may include crowd-dense data 440 of at least one preset area, and the output may include a second embedded feature 457 .

[0212] Similar to the first embedding feature, the second embedding feature can characterize the characteristics of people-dense data.

[0213] In some embodiments, the input of the embedding feature extraction layer also includes a regional environment of at least one preset region. For more description of the regional environment of the preset region, see Figure 3 Related content.

[0214] In some embodiments of the present specification, since people-intensive data is relatively important and low-noise compared to visual modal information, auditory modal information, tactile modal information, monitoring personnel information and order information, the people-intensive data is processed separately through the embedded feature extraction layer to determine the second embedded feature, rather than processing multiple information through the modal integration layer. This can ensure the original appearance of the second embedded feature of the people-intensive data as much as possible, which is beneficial to determining the accuracy of the conflicting information determined by the subsequent output layer.

[0215] In some embodiments, the output layer 455 processes the first embedded feature 454 and the second embedded feature 457 to obtain conflict information 460. For more information about conflict information, see Figure 3 Related content.

[0216] The output layer 455 refers to a model layer structure for obtaining conflict information. In some embodiments, the output layer 455 can be a machine learning model, such as a neural network model.

[0217] The input of the output layer 455 may include the first embedded feature 454 and the second embedded feature 457 , and the output may include the conflict information 460 .

[0218] In some embodiments, the output of the modality integration layer 453 and the output of the embedding feature extraction layer 456 - 1 may be the input of the output layer 455 , and the modality integration layer 453 , the embedding feature extraction layer 456 - 1 and the output layer 455 may be jointly trained.

[0219] In some embodiments, each group of fifth training samples of the joint training includes sample visual modal information, sample auditory modal information, sample tactile modal information, sample monitoring personnel information, sample order information and sample personnel density data, and the fifth label is the actual conflict information corresponding to the preset area of ​​the group of fifth training samples (such as conflicts between store clerks and customers or conflicts between customers, etc.).

[0220] In some embodiments, the processor can input sample visual modal information, sample auditory modal information, sample tactile modal information, sample monitoring personnel information and sample order information into the initial modal integration layer to obtain the output of the initial modal integration layer; input the sample personnel intensive data into the initial embedding feature extraction layer to obtain the output of the initial embedding feature extraction layer. The output of the initial modal integration layer and the output of the initial embedding feature extraction layer are used as the data of the fifth training sample, input into the output layer, and the predicted conflict information output by the output layer is obtained. A loss function is constructed based on the actual conflict information of the fifth training sample and the predicted conflict information output by the output layer, and the parameters of the modal integration layer, the embedding feature extraction layer and the output layer are updated synchronously. Through parameter updating, the trained modal integration layer, the embedding feature extraction layer and the output layer are obtained.

[0221] The sample visual modal information, the sample auditory modal information, and the sample tactile modal information may be the output of the visual modal layer in the historical data, the output of the auditory modal layer in the historical data, and the output of the tactile modal layer in the historical data, respectively. The sample monitoring personnel information, the sample order information, and the sample personnel-intensive data may be the monitoring personnel information, the order information, and the personnel-intensive data in the historical data. The fifth label is the conflict information of the fifth training sample of the group. The sample vibration data may be the vibration data of the store in the historical data. In some embodiments, the fifth label may be determined based on the actual conflict information of the historical data.

[0222] In some embodiments, when the input of the embedding feature extraction layer includes a regional environment of a preset area, the fifth training sample also includes a sample regional environment.

[0223] In some embodiments of the present specification, by processing different types of data in different layers separately, fusing features through the modal integration layer, extracting features of densely populated data through the embedded feature extraction layer, and then predicting through the output layer, it is possible to efficiently process multiple types of input data, which is conducive to quickly and accurately determining conflict information, allowing staff of large catering stores to deal with conflicts in the first time, thereby improving the management capabilities of large catering stores.

[0224] Some embodiments of the present specification also provide an intelligent store conflict identification device, which includes at least one processor and at least one memory, wherein the at least one memory is used to store computer instructions, and the at least one processor is used to execute at least part of the computer instructions to implement any of the intelligent store conflict identification methods described in the embodiments of the present specification.

[0225] Some embodiments of the present specification also provide a computer-readable storage medium, wherein the storage medium stores computer instructions. When a computer reads the computer instructions in the storage medium, the computer executes the smart store conflict identification method described in any one of the embodiments of the present specification.

[0226] The basic concepts have been described above. Obviously, for those skilled in the art, the above detailed disclosure is only for example and does not constitute a limitation of this specification. Although not explicitly stated here, those skilled in the art may make various modifications, improvements and corrections to this specification. Such modifications, improvements and corrections are suggested in this specification, so such modifications, improvements and corrections still belong to the spirit and scope of the exemplary embodiments of this specification.

[0227] In addition, unless explicitly stated in the claims, the order of the processing elements and sequences described in this specification, the use of alphanumeric characters, or the use of other names are not intended to limit the order of the processes and methods of this specification. Although the above disclosure discusses some invention embodiments that are currently considered useful through various examples, it should be understood that such details are only for illustrative purposes, and the attached claims are not limited to the disclosed embodiments. On the contrary, the claims are intended to cover all modifications and equivalent combinations that are consistent with the essence and scope of the embodiments of this specification. For example, although the system components described above can be implemented by hardware devices, they can also be implemented only by software solutions, such as installing the described system on an existing server or mobile device.

[0228] Similarly, it should be noted that in order to simplify the description disclosed in this specification and thus help understand one or more embodiments of the invention, in the above description of the embodiments of this specification, multiple features are sometimes combined into one embodiment, figure or description thereof. However, this disclosure method does not mean that the features required by the subject matter of this specification are more than the features mentioned in the claims. In fact, the features of the embodiments are less than all the features of the single embodiment disclosed above.

[0229] In some embodiments, numbers describing the number of components and attributes are used. It should be understood that such numbers used in the description of the embodiments are modified by the modifiers "about", "approximately" or "substantially" in some examples. Unless otherwise specified, "about", "approximately" or "substantially" indicate that the numbers are allowed to vary by ±20%. Accordingly, in some embodiments, the numerical parameters used in the specification and claims are approximate values, which may change according to the required features of individual embodiments. In some embodiments, the numerical parameters should take into account the specified significant digits and adopt the general method of retaining digits. Although the numerical domains and parameters used to confirm the breadth of their range in some embodiments of this specification are approximate values, in specific embodiments, the setting of such numerical values ​​is as accurate as possible within the feasible range.

[0230] Finally, it should be understood that the embodiments described in this specification are only used to illustrate the principles of the embodiments of this specification. Other variations may also fall within the scope of this specification. Therefore, as an example and not a limitation, alternative configurations of the embodiments of this specification may be considered consistent with the teachings of this specification. Accordingly, the embodiments of this specification are not limited to the embodiments explicitly introduced and described in this specification.

Claims

1. An intelligent store conflict identification system, characterized in that: The system includes an infrared thermal imager, an audio acquisition device, an alarm system, a communication device and a processor; The infrared thermal imager is arranged in a preset area of ​​the store, and the infrared thermal imager is configured to obtain infrared thermal image data of the preset area; The audio acquisition device is disposed in the preset area of ​​the store, and the audio acquisition device is configured to acquire audio data of the preset area; The alarm system is configured to send conflict information to the responsible store clerk and execute an alarm based on the alarm instruction; The communication device is configured to implement data communication between the infrared thermal imager, the audio acquisition device, the alarm system and the processor; The processor is configured to: Determining the conflict information based on the infrared thermal imaging data and the audio data; Determining the responsible clerk based on the conflict information; as well as Based on the responsible clerk and the conflict information, the alarm instruction is determined and sent to the alarm system.

2. The system according to claim 1, characterized in that The system further comprises an order monitoring device; the order monitoring device is configured to monitor order information of a table; The processor is further configured to: Determine monitoring personnel information based on the infrared thermal imaging data and the order information; The conflict information is determined based on the monitoring personnel information and the audio data.

3. The system according to claim 2, characterized in that The system also includes a network monitoring device and a vibration sensor; The network monitoring device is arranged in the preset area of ​​the store, and the network monitoring device is configured to monitor and obtain the crowd density data of the preset area; The vibration sensor is disposed at a preset position in the preset area, and the vibration sensor is configured to monitor vibration data at the preset position; The processor is further configured to: Based on the monitoring personnel information, the audio data, the vibration data and the personnel density data, the conflict information is determined by a conflict prediction model, and the conflict prediction model is a machine learning model.

4. The system according to claim 1, characterized in that The processor is further configured to: Based on the conflict information, determining the conflict risk of the preset area; Based on the conflict risk, the responsible clerk is determined.

5. The system according to claim 4, characterized in that The processor is further configured to: Determining a risk change trend based on the conflict risk and the regional environment of the preset area; Based on the risk change trend, the responsible clerk is determined.

6. A smart store conflict identification method, characterized in that: The method comprises: Acquire infrared thermal imaging data of a preset area and audio data of the preset area; Determining conflict information based on the infrared thermal imaging data and the audio data; Determining a responsible clerk based on the conflict information; and Based on the responsible clerk and the conflict information, an alarm instruction is determined and sent to an alarm system.

7. The method according to claim 6, characterized in that The determining of conflict information based on the infrared thermal imaging data and the audio data includes: Determine monitoring personnel information based on the infrared thermal imaging data and order information; The conflict information is determined based on the monitoring personnel information and the audio data.

8. The method according to claim 7, characterized in that The determining the conflict information based on the monitoring personnel information and the audio data includes: Based on the monitoring personnel information, the audio data, the vibration data and the personnel density data, the conflict information is determined by a conflict prediction model, and the conflict prediction model is a machine learning model.

9. The method according to claim 6, characterized in that Determining the responsible clerk based on the conflict information includes: Based on the conflict information, determining the conflict risk of the preset area; Based on the conflict risk, the responsible clerk is determined.

10. A computer-readable storage medium, characterized in that: The storage medium stores computer instructions. When a computer reads the computer instructions in the storage medium, the computer executes the method according to any one of claims 6 to 9.

Images