An image recognition method and a management device

Abstract

The present invention provides an image recognition method and a management device. The management device has multiple shared memories. After a set period arrives, it respectively sends first control information to N front-end devices to instruct the front-end devices to store the captured image frames in the first shared memory. After the next set period arrives, it obtains the image frames from the first shared memory for recognition, and respectively sends second control information to M front-end devices among the N front-end devices to instruct the front-end devices to store the captured image frames in the second shared memory. This solution sets at least two shared memories inside the management device and combines with the set period, so that for one set period, the management device reads the image frames written by the front-end devices from one shared memory for recognition processing, and allows the front-end devices to write the image frames into at least one other shared memory, thereby separating the reading and writing of the shared memory and improving the processing efficiency of the management device.

Description

Technical Field

[0001] The embodiments of the present invention relate to the technical field of artificial intelligence, and in particular, to an image recognition method and a management device. Background Art

[0002] With the continuous development of artificial intelligence and deep learning technologies, people are no longer satisfied with simply controlling machines in traditional ways. Thus, the use of gesture commands has emerged. For example, the popularity of motion-sensing games and the gesture control of smart home devices all utilize the user's gestures or body postures to determine the user's intention and convey the user's commands.

[0003] In the context of the home scenario, smart TVs, smart speakers, smart refrigerators, home smart monitoring devices, etc. all use gesture commands for control, mainly through built-in smart chips to help users achieve device control. However, due to the addition of smart chips, these smart home devices are much more expensive than ordinary household appliances on the market, and in addition, it is not conducive to unified control and management.

[0004] To solve the above problems, various manufacturers have begun to focus on the research and development of smart home butlers, hoping that the developed smart home butlers can communicate with various smart home devices, obtain the image frames reported by the various smart home devices, and the smart home butlers can uniformly manage these smart home devices, including data preprocessing, face recognition, and gesture recognition, so as to realize the recognition of the user's gestures or body postures, enabling the smart home devices to respond to the user's operation instructions.

[0005] Therefore, there is an urgent need for a solution to improve the processing speed of smart home butlers, so as to timely respond to user operations through smart home devices. Summary of the Invention

[0006] The present invention provides an image recognition method and a management device to improve the processing speed of smart home butlers, so as to timely respond to user operations through smart home devices.

[0007] In a first aspect, an embodiment of the present invention provides an image recognition method, which is applied to a management device provided with a plurality of shared memories; the method includes: after a set period arrives, respectively sending first control information to N front-end devices; the first control information is used to instruct the front-end devices to store the captured image frames in a first shared memory; after the next set period arrives, obtaining the image frames from the first shared memory for recognition, and respectively sending second control information to M front-end devices among the N front-end devices; the second control information is used to instruct the front-end devices to store the captured image frames in a second shared memory; the first shared memory and the second shared memory are different shared memories among the plurality of shared memories; M is not greater than N.

[0008] Based on this solution, by setting at least two shared memories inside the management device and combining with a set period, for a set period, the management device reads the image frames written by the front-end devices from one shared memory for recognition processing, and allows the front-end devices to write the image frames into at least one other shared memory, so as to separate the reading and writing of the shared memory and improve the processing efficiency of the management device. In addition, through the setting of the set period, when the set period arrives, the management device triggers the processing of the image frames in the shared memory without waiting for each front-end device to write its respective image frames, which further ensures that the management device can still perform subsequent recognition processing even if a certain front-end device is abnormal.

[0009] In a possible implementation method, the first control information is further used to instruct the front-end device to send a first response message after storing the image frame in the first shared memory; the method further includes: before the set period arrives, if the first response message sent by the i-th front-end device is received, the second control information is sent to the i-th front-end device; where the i-th front-end device is any one of the N front-end devices.

[0010] Based on this solution, for a set period, when the front-end devices send image frames to the management device, if a front-end device has already uploaded the image frame to the first shared memory and sent a first response message to the management device before the set period ends, the management device can send a second control message to this front-end device when receiving the first response message, instead of waiting until the end of the set period when the management device sends the respective second control information to multiple front-end devices at the same time. This method can better improve the efficiency of the front-end devices when writing image frames and can avoid the problem of loss of one or some second control information that may be caused by sending the second control information simultaneously.

[0011] In a possible implementation method, the sending of the second control information to M front-end devices among the N front-end devices respectively includes: before the set period arrives, determining that the first response messages sent by the M front-end devices have not been received; after the set period arrives, sending the second control information to the M front-end devices respectively; the second control information is used to instruct the front-end device to retransmit the image frames that have not been stored in the first shared memory to the second shared memory.

[0012] Based on this solution, for a set period, when a front-end device sends an image frame to a management device, if the front-end device fails to report and store the image frame in the first shared memory as scheduled within this set period due to its own anomalies or network anomalies, etc., then the management device can send second control information to the front-end device at the moment when this set period arrives. At this time, the second control message is used to instruct the front-end device to continue uploading the image frames that were not successfully stored in the first shared memory in the previous period to the second shared memory in the next set period. In this way, the processing of all image frames collected by the front-end device can be achieved.

[0013] In a possible implementation method, the first control information is further used to instruct the front-end device to send a first response message after storing the image frame in the first shared memory; M = N; the sending of the second control information to M of the N front-end devices respectively includes: sending the second control information to the j-th front-end device; where the j-th front-end device is the front-end device among the N front-end devices that has not successfully sent the first response message; the second control information is used to instruct the j-th front-end device to retransmit the un-stored image frames collected by it to the second shared memory; sending the second control information to the k-th front-end device; where the k-th front-end device is the front-end device among the N front-end devices that has successfully sent the first response message; the second control information is used to instruct the k-th front-end device to store the subsequent image frames of the image frames that have been stored in the first shared memory in the second shared memory.

[0014] Based on this solution, for a set period, if a front-end device fails to complete the upload of the image frame as scheduled within this set period, then the management device can send second control information to the front-end device(s) at the moment when this set period arrives to inform the front-end device(s) to retransmit the un-stored image frames collected by it to the second shared memory; if a front-end device can complete the upload of the image frame as scheduled within this set period, then the second control information sent by the management device to the front-end device(s) is used to instruct it to store the subsequent image frames in the second shared memory. In this way, reasonable control of front-end devices in different data upload states can be achieved.

[0015] In a possible implementation method, the obtaining of the image frame from the first shared memory for recognition includes: setting N processes, each process being used to recognize the image frames collected by a corresponding front-end device; each process synchronously recognizes the respective image frames obtained from the first shared memory.

[0016] Based on this solution, for the image frames reported by each front-end device and already stored in the shared memory of the management device, by setting an identification process for each image frame reported by the front-end device, when the set period arrives, each identification process can be enabled to identify the image frames reported by multiple front-end devices simultaneously, greatly improving the identification speed.

[0017] In a possible implementation method, the set period is determined according to the identification duration of identifying one image frame; and / or the first control information is used to instruct the front-end device to store a captured image frame in the first shared memory.

[0018] Based on this solution, by setting an identification process for each image frame reported by the front-end device and determining the identification duration of the management device for simultaneously processing one image frame sent by each of the N front-end devices, and using this identification duration as the set period, the management device can successfully identify any number of image frames not greater than N frames within this set period.

[0019] In a possible implementation method, the obtaining of the image frame from the first shared memory for identification includes: identifying the front-end device that sent the first response message according to the received first response message; and obtaining the image frame from the first shared memory for identification according to the identification.

[0020] Based on this solution, for a set period, if the front-end device successfully uploads an image frame to the management device, when the management device receives the first response message sent from the front-end device, the corresponding front-end device that sent the first response message is identified. This facilitates the management device to determine which front-end device(s) successfully uploaded the image frame within this set period according to this identification, so that the management device can obtain the image frame from the preset position stored in the first shared memory, accelerating the overall process of image recognition.

[0021] In a second aspect, an embodiment of the present invention provides a management device, which is configured with multiple shared memories; further includes: a transceiver, configured to send first control information to N front-end devices respectively after the set period arrives; the first control information is used to instruct the front-end device to store the captured image frame in the first shared memory; a processor, configured to obtain the image frame from the first shared memory for identification after the next set period arrives; the transceiver is further configured to send second control information to M front-end devices among the N front-end devices respectively; the second control information is used to instruct the front-end device to store the captured image frame in the second shared memory; the first shared memory and the second shared memory are different shared memories among the multiple shared memories; M is not greater than N.

[0022] Based on this solution, by setting at least two shared memories inside the management device and combining with a set period, for a set period, the management device reads the image frames written by the front-end devices from one shared memory for recognition processing, and allows the front-end devices to write the image frames into at least one other shared memory, so as to separate the reading and writing of the shared memory and improve the processing efficiency of the management device. In addition, through the setting of the set period, when the set period arrives, the management device triggers the processing of the image frames in the shared memory without waiting for each front-end device to write its respective image frames, which further ensures that the management device can still perform subsequent recognition processing even if a certain front-end device is abnormal.

[0023] In a possible implementation method, the first control information is further used to instruct the front-end device to send a first response message after storing the image frame in the first shared memory; the transceiver is further configured to: before the set period arrives, if it receives the first response message sent by the i-th front-end device, send the second control information to the i-th front-end device; where the i-th front-end device is any one of the N front-end devices.

[0024] Based on this solution, for a set period, when the front-end devices send image frames to the management device, if a front-end device has uploaded the image frame to the first shared memory and sent a first response message to the management device before the set period ends, the management device can send the second control message to this front-end device when it receives the first response message, instead of waiting until the end of the set period when the management device sends the respective second control information to multiple front-end devices at the same time. This method can better improve the efficiency of the front-end devices when writing image frames and avoid the problem of loss of some second control information that may be caused by sending the second control information simultaneously.

[0025] In a possible implementation method, the processor is configured to: before the set period arrives, determine that it has not received the first response messages sent by the M front-end devices; the transceiver is configured to: after the set period arrives, send the second control information to the M front-end devices respectively; the second control information is used to instruct the front-end devices to retransmit the image frames that have not been stored in the first shared memory to the second shared memory.

[0026] Based on this solution, for a set period, when a front-end device sends an image frame to a management device, if the front-end device fails to report and store the image frame in the first shared memory within the set period due to its own anomalies or network anomalies, etc., then the management device can send second control information to the front-end device at the moment when the set period arrives. At this time, the second control message is used to instruct the front-end device to continue uploading the image frames that were not successfully stored in the first shared memory in the previous period to the second shared memory in the next set period. In this way, all the image frames collected by the front-end device can be processed.

[0027] In a possible implementation method, the first control information is further used to instruct the front-end device to send a first response message after storing the image frame in the first shared memory; M = N; the transceiver is configured to: send second control information to the j-th front-end device, where the j-th front-end device is the front-end device among the N front-end devices that has not successfully sent the first response message; the second control information is used to instruct the j-th front-end device to retransmit the un-stored image frames collected by it to the second shared memory; send second control information to the k-th front-end device, where the k-th front-end device is the front-end device among the N front-end devices that has successfully sent the first response message; the second control information is used to instruct the k-th front-end device to store the subsequent image frames of the image frames that have been stored in the first shared memory in the second shared memory.

[0028] Based on this solution, for a set period, if a front-end device fails to complete the upload of an image frame as scheduled in the set period, then the management device can send second control information to the front-end device(s) at the moment when the set period arrives to inform the front-end device(s) to retransmit the un-stored image frames collected by it to the second shared memory; if a front-end device can complete the upload of an image frame as scheduled in the set period, then the second control information sent by the management device to the front-end device(s) is used to instruct it to store the subsequent image frames in the second shared memory. In this way, reasonable control of front-end devices with different data upload statuses can be achieved.

[0029] In a possible implementation method, the processor is configured to: set N processes, each process is used to identify the image frames collected by a corresponding front-end device; each process synchronously identifies the respective image frames obtained from the first shared memory.

[0030] Based on this solution, for the image frames reported by each front-end device and already stored in the shared memory of the management device, by setting an identification process for each image frame reported by the front-end device, when the set period arrives, each identification process can be enabled to identify the image frames reported by multiple front-end devices simultaneously, greatly improving the identification speed.

[0031] In a possible implementation method, the set period is determined according to the identification duration of identifying one image frame; and / or the first control information is used to instruct the front-end device to store a frame of the collected image in the first shared memory.

[0032] Based on this solution, by setting an identification process for each image frame reported by the front-end device and determining the identification duration of the management device for simultaneously processing one image frame sent by each of the N front-end devices, and using this identification duration as the set period, the management device can successfully identify any number of image frames not greater than N frames within this set period.

[0033] In a possible implementation method, the processor is configured to: identify the front-end device that sends the first response message according to the received first response message; obtain the image frame from the first shared memory according to the identification for identification.

[0034] Based on this solution, for a set period, if the front-end device successfully uploads an image frame to the management device, when the management device receives the first response message sent from the front-end device, the corresponding front-end device that sends the first response message is identified, so that it is convenient for the management device to determine which specific front-end device(s) successfully uploaded the image frame within this set period according to this identification, and thus the management device can obtain the image frame from the preset position in the first shared memory, accelerating the overall process of image recognition.

[0035] In a third aspect, an embodiment of the present invention provides a computing device, including:

[0036] A memory for storing program instructions;

[0037] A processor for calling the program instructions stored in the memory and executing any implementation method of the first aspect according to the obtained program.

[0038] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, where the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to cause a computer to execute any implementation method of the first aspect. Description of the Drawings

[0039] To more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the accompanying drawings required for the description of the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can be obtained based on these drawings.

[0040] Figure 1 A possible schematic diagram of the system architecture provided by the embodiments of the present invention;

[0041] Figure 2 An image recognition method provided by the embodiments of the present invention;

[0042] Figure 3 A schematic diagram of the image recognition process provided by the embodiments of the present invention;

[0043] Figure 4 A management device provided by the embodiments of the present invention. Detailed implementation manners

[0044] In order to make the objectives, technical solutions, and advantages of the present invention clearer, the following will further describe the present invention in detail with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0045] As Figure 1 shown, a possible schematic diagram of the system architecture provided by the embodiments of the present invention, the system includes a management device 110 and at least one front-end device 120 (such as Figure 1 the front-end device 121, front-end device 122, and front-end device 123 shown in

[0046] Among them, the front-end device 120 is configured with a video acquisition unit, such as a camera. After the front-end device 120 is powered on, the video acquisition unit therein starts to work, which may include collecting status data within a preset range, forming frames of image frames, and uploading the collected image frames to the management device 110, and the management device 110 processes the image frames collected by the front-end device 120. In addition, the front-end device 120 can also respond to the control information sent by the management device 110, including the position information for storing the collected image frames in the management device 110, and after the management device 110 finishes processing the image frames, if the front-end device 120 needs to make a response behavior, the control information can also include response information.

[0047] The management device 110 is configured with multiple shared memories for storing the image frames collected by the front-end device 110. The management device 110 can communicate with the front-end device 120 in a wired or wireless manner, including sending control information to the front-end device 120 to indicate the specific location information where the front-end device 120 stores the collected image frames, such as which shared memory to store in. And when the management device 110 determines that the front-end device 120 needs to perform a response behavior after processing the image frames, the management device 110 can also send a response message to the front-end device 120 to indicate the front-end device 120 to perform the corresponding response behavior.

[0048] Among them, the management device 110 and the front-end device 120 can be located in the same area or in different areas, which is not limited in the present invention; at least one front-end device 120 can be located in the same area or in different areas, which is not limited in the present invention.

[0049] The above system architecture can be applied to many scenarios such as the smart home scenario, the indoor / outdoor video surveillance scenario, etc. In the embodiment of the present invention, the smart home scenario will be taken as an example for description.

[0050] In the smart home scenario, there is a smart home butler and at least one smart home device. The smart home device can be devices such as a smart TV, a smart speaker, a smart refrigerator, and a home smart monitor. Among them, the smart home butler is used to centrally process the image frames sent by at least one smart home device. There may be user gesture commands or body posture commands in the image frames to be recognized. And after recognizing that the user gesture command or body posture command conforms to the preset control command for the smart home device, the smart home butler can send a feedback message to the corresponding smart home device to indicate the smart home device to perform the corresponding response behavior. In the above-described smart home scenario, the management device is specifically the smart home butler, and the front-end device is specifically any smart home device that can communicate with the smart home butler.

[0051] At the current stage, the smart home butler adopts the technology of TensorRT. For example, it uses the multi-batch function in TensorRT version 6.0 to process image frames sent by multiple smart home devices simultaneously. During specific implementation, the data processing capacity in TensorRT version 6.0, that is, batchsize, can be configured according to the number of smart home devices actually connected to the smart home butler. For example, in a household, there are 6 smart home devices, and these 6 smart home devices are all connected to the same smart home butler. In order to improve the processing efficiency of the image frames collected by the smart home butler for these 6 smart home devices, the multi-batch function in TensorRT version 6.0 can be used to set batchsize to 6, so that the smart home butler can process the image frames collected by these 6 smart home devices simultaneously.

[0052] However, there are some deficiencies in the above technology. For example, during the implementation of the multi-batch function in TensorRT version 6.0, it is necessary to wait until the smart home butler has received all the image frames uploaded by these 6 smart home devices before it can process multiple image frames simultaneously. However, due to some abnormal events, the smart home butler will be in a state of waiting to process data for a long time, which will cause some smart home devices that upload image frames in a timely manner to be unable to respond to the user's operation instructions in a timely manner, greatly affecting the user experience. Abnormal events can include the damage of one or some smart home devices among multiple smart home devices, resulting in the inability to upload image frames in a timely manner, or the inability of one or some smart home devices to upload image frames in a timely manner due to network failures or other reasons.

[0053] Based on Figure 1 the system architecture shown above and the above reasons, an embodiment of the present invention provides an image recognition method, which is executed by Figure 1 the management device shown. As Figure 2 shown, the method includes the following steps:

[0054] Step 201, after the set period arrives, send the first control information to N front-end devices respectively.

[0055] In this step, the first control information is used to instruct the front-end device to store the collected image frames in the first shared memory.

[0056] Step 202, after the next set period arrives, obtain the image frames from the first shared memory for recognition, and send the second control information to M front-end devices among the N front-end devices respectively.

[0057] In this step, the second control information is used to instruct the front-end device to store the captured image frames in the second shared memory; the first shared memory and the second shared memory are different shared memories among the multiple shared memories; M is not greater than N.

[0058] Based on this solution, by setting at least two shared memories inside the management device and combining with a set period, for a set period, the management device reads the image frames written by the front-end device from one shared memory for recognition processing, and allows the front-end device to write the image frames into at least one other shared memory, so as to separate the reading and writing of the shared memory and improve the processing efficiency of the management device. In addition, through the setting of the set period, when the set period arrives, the management device triggers the processing of the image frames in the shared memory without waiting for each front-end device to write its respective image frames, which further ensures that the management device can continue with subsequent recognition processing without being affected by an abnormal front-end device.

[0059] The following will separately describe the above steps in detail with examples.

[0060] In the embodiment of the present invention, TensorRT version 7.0 and above is selected as the technology for image recognition. Since the construction process of the TensorRT network engine used in the image recognition process is complex and time-consuming, before step 201 is implemented, the TensorRT network engine can be pre-constructed. In the embodiment of the present invention, 6 front-end devices are connected to the same management device, and based on the premise that the management device can simultaneously recognize one image frame uploaded by each of the 6 front-end devices, the value of batchsize in the pre-constructed TensorRT network engine can be set to an integer multiple of 6, such as 6, 12, 18, etc. After the TensorRT network engine is successfully constructed, it can be serialized and saved to a.txt format document for reading. When the image is to be recognized, the saved TensorRT network engine can be deserialized from the.txt document.

[0061] It should be noted that, compared with the previous versions, TensorRT version 7.0 and above can support images with any number of frames as input, as long as the number of input image frames is not greater than the value of batchsize set when the TensorRT network engine is pre-constructed.

[0062] In an implementation of the above step 201, the set period is determined according to the recognition duration of recognizing one image frame.

[0063] For the example where the aforementioned 6 front-end devices are connected to the same management device, determine the recognition duration for the management device to simultaneously recognize one frame of image uploaded by each of the 6 front-end devices. For example, it is 130 milliseconds, so 130 milliseconds can be directly used as the set period. Further, considering the adequacy of the recognition duration, the set period can be appropriately set larger. For example, 150 milliseconds can be used as the set period. In the embodiment of the present invention, taking 150 milliseconds as the set period as an example for illustration, that is, within 150 milliseconds, the management device can successfully recognize 6 frames of images.

[0064] It should be noted that the set period can also be determined according to the recognition duration of recognizing adjacent multiple frames of images. For example, in the aforementioned example where 6 front-end devices are connected to the same management device, determine the recognition duration for the management device to simultaneously recognize two adjacent frames of images uploaded by each of the 6 front-end devices. For example, it is 280 milliseconds. Further, considering the adequacy of the recognition duration, the set period can be appropriately set larger. For example, 300 milliseconds can be used as the set period. Of course, at this time, the value of batchsize needs to be adjusted. For example, the value of batchsize can be set to 12.

[0065] In the embodiment of the present invention, based on the process of simultaneously recognizing one frame of image sent by each of the 6 front-end devices, the value of batchsize set is 6, and the set period is 150 milliseconds.

[0066] In an implementation of the above step 201, at the start moment of a set period, that is, at the end moment of the previous set period, the management device sends the first control information to N front-end devices respectively to instruct the front-end devices that receive the first control information to store the captured image frames in the first shared memory of the management device.

[0067] For example, the management device is configured with 3 shared memories, denoted as shared memory 1, shared memory 2, and shared memory 3 respectively. Among them, for each shared memory, it can be further divided into multiple preset positions. For these multiple preset positions, mapping relationships can be established with multiple front-end devices in advance to identify that the front-end devices can store the captured image frames in the corresponding preset positions. Thus, at the start moment of a set period, the management device determines that one of the shared memories, such as shared memory 1, already stores the image frames of the previous set period and these image frames are about to be used in the operation process of image recognition. Therefore, the management device can send the first control information to N front-end devices respectively. For example, the first control information can be used to instruct the N front-end devices to store the captured image frames in shared memory 2 or shared memory 3.

[0068] In an implementation of the above step 201, the first control information is further used to instruct the front-end device to send a first response message to the management device after storing the image frame in the first shared memory. Among the N front-end devices, when the first control information sent by the management device is received, if there is a front-end device i that has sent a first response message to the management device before the current set period arrives, at this time, the management device can send second control information to the front-end device i at the moment of receiving the first response message, so as to instruct the front-end device i to store the captured image frame in the second shared memory, instead of waiting until the current set period arrives to send the second control information to the front-end device i. Therefore, it can better improve the efficiency of the front-end device when writing the image frame and avoid the loss problem of some second control information that may be caused by sending the second control information simultaneously.

[0069] In an implementation of the above step 201, when a set period arrives, if the management device fails to receive the first response messages sent by M front-end devices among the N front-end devices as scheduled, it means that there may be some abnormal reasons for these M front-end devices, resulting in their inability to upload the image frames that should belong to the current set period to the first shared memory in time. Therefore, considering data integrity, the management device will send second control information to these M front-end devices at the moment when the current set period arrives. The second control information is used to instruct these M front-end devices to re-upload the image frames that were not uploaded as scheduled in the current set period to the second shared memory in the next set period.

[0070] In an implementation of the above step 201, during a set period, some front-end devices may send the first response message to the management device before the current set period arrives or exactly when the current set period arrives, and some front-end devices may not be able to send the first response message to the management device at the moment when the current set period arrives at the latest. Based on the above two situations, the management device can make the following response actions:

[0071] For the first situation, the management device can send second control information to the front-end device that sent the first response message at the moment of receiving the first response message or at the end of the current set period. The second control information is used to instruct the corresponding front-end device to upload the subsequent image frames of the image frames that have been uploaded in the current set period to the second shared memory in the next set period.

[0072] For the second case, at the end of the current setting cycle, the management device may send second control information to the front-end devices that have not sent the first response message during the current setting cycle. This second control information is used to instruct the corresponding front-end devices to re-upload to the second shared memory during the next setting cycle the image frames that could not be uploaded to the first shared memory as scheduled during the current setting cycle.

[0073] In an implementation of step 202 above, at the start of a setting cycle, the management device enables N processes. Each of these N processes is used to identify a frame of image captured by a preset front-end device. After the N processes are enabled, each process simultaneously obtains a corresponding frame of image from a preset location in the first shared memory for identification.

[0074] In an implementation of step 202 above, during a setting cycle, regardless of whether the management device receives the first response message from the front-end device at any moment before the end of the current setting cycle or at the end of the current setting cycle, the management device will identify the front-end device that sent the first response message. For example, before the front-end device sends the first response message, the management device records the flag bit corresponding to this front-end device as 0; when the management device receives the first response message sent by this front-end device, the management device will change the flag bit corresponding to this front-end device. For example, it can record the identification bit corresponding to this front-end device as 1. Thus, at the end of the current setting cycle, that is, at the start of the next setting cycle, the management device can, based on the information of the flag bit, enable the identification process corresponding to the front-end device with the flag bit of 1, and let the identification process obtain the image frame from the preset location in the first shared memory for identification, without enabling the identification process corresponding to the front-end device with the flag bit of 0, so as to accurately implement the reading of the image frames uploaded by the front-end devices to the first shared memory.

[0075] Further, after the identification process completes the identification of the image frame, the management device can reset the flag bit corresponding to the front-end device to 0 again, so as to facilitate using the flag bit to identify the front-end devices that have sent the first response message and those that have not sent the first response message in the next setting cycle.

[0076] It should be noted that in the embodiments of the present invention, the first control information, the second control information, and the first response message can be transmitted through websocket instructions.

[0077] Such as Figure 3As shown in the figure, it is a schematic diagram of an image recognition process provided by an embodiment of the present invention. Among them, the main process and the shared memory are both set in the management device. The main process has the following functions: on the one hand, it can be used to send websocket instructions to multiple front-end devices at the arrival time of the set period, for instructing the front-end devices to store the image frames in either shared memory 1 or shared memory 2; on the other hand, it can also be used to obtain the image frames from the shared memory that has stored the image frames at the arrival time of the set period, and perform face detection and gesture recognition on them. Each of the N front-end devices is used to store the collected image frames in the corresponding shared memory according to the received websocket instructions.

[0078] Reference Figure 3 , in the main process, a set count value T is set. When Ttmp = 0, the main process sends a websocket instruction to each front-end device, notifying it to store a frame of the collected image frames in shared memory 1. After each front-end device finishes storing the image frames in shared memory 1, it sends a websocket instruction to the main process, notifying the main process that the front-end device has finished storing. At the same time, the main process will set the data storage flag bit corresponding to the front-end device to 1.

[0079] When Ttmp = T, the main process will send a websocket instruction to each front-end device, notifying it to store the next frame of the collected image frames in shared memory 2. At the same time, the main process obtains the complete image frames from shared memory 1 according to the setting of the data storage flag bits of each front-end device. If there are still front-end devices that have not completed data storage when Ttmp = T, the main process needs to send a websocket instruction to the corresponding front-end device, notifying it to stop the current storage and instead store it in shared memory 2.

[0080] When T < Ttmp < 2T, each front-end device stores the collected image frames in shared memory 2. At this time, the main process will obtain the image frames from shared memory 1 and set the data storage flag bits of each front-end device to 0. The obtained image frames can be put into the next process for face detection and gesture recognition.

[0081] When Ttmp = 2T, the value of Ttmp is cleared, and the main process will send a websocket instruction to each front-end device, notifying it to store the next frame of video data in shared memory 1. At the same time, according to the setting of the data storage flag bits of each front-end device, it obtains the complete image frames from shared memory 2. If there are still front-end devices that have not completed data storage when Ttmp = 2T, the main process needs to send a websocket instruction to the corresponding front-end device, notifying it to stop the current storage and instead store it in shared memory 1.

[0082] Based on the same concept, an embodiment of the present invention provides a management device, such as Figure 4 as shown, the management device is configured with a plurality of shared memories, and further includes a transceiver 401 and a processor 402:

[0083] The transceiver 401 is used to send first control information to N front-end devices respectively after a set period arrives; the first control information is used to instruct the front-end devices to store the captured image frames in the first shared memory.

[0084] The processor 402 is used to obtain the image frames from the first shared memory for recognition after the next set period arrives.

[0085] The transceiver 401 is further used to send second control information to M front-end devices among the N front-end devices respectively; the second control information is used to instruct the front-end devices to store the captured image frames in the second shared memory; the first shared memory and the second shared memory are different shared memories among the plurality of shared memories; M is not greater than N.

[0086] Further, for this management device, the first control information is further used to instruct the front-end devices to send a first response message after storing the image frames in the first shared memory; the transceiver 401 is further used to, before the set period arrives, if receiving the first response message sent by the i-th front-end device, send the second control information to the i-th front-end device; where the i-th front-end device is any one of the N front-end devices.

[0087] Further, for this management device, the processor 402 is specifically used to determine that the first response messages sent by the M front-end devices have not been received before the set period arrives; the transceiver 401 is specifically used to send the second control information to the M front-end devices respectively after the set period arrives; the second control information is used to instruct the front-end devices to retransmit the image frames that have not been stored in the first shared memory to the second shared memory.

[0088] Further, for the management device, the first control information is further used to instruct the front-end device to send a first response message after storing the image frame in the first shared memory; the transceiver 401 is specifically configured to send second control information to the j-th front-end device, where the j-th front-end device is the front-end device among the N front-end devices that has not successfully sent the first response message; the second control information is used to instruct the j-th front-end device to retransmit the un-stored image frame collected by the j-th front-end device to the second shared memory; send second control information to the k-th front-end device, where the k-th front-end device is the front-end device among the N front-end devices that has successfully sent the first response message; the second control information is used to instruct the k-th front-end device to store the subsequent image frames of the image frames already stored in the first shared memory in the second shared memory.

[0089] Further, for the management device, the processor 402 is specifically configured to set N processes, each process is used to identify the image frames collected by the corresponding front-end device; the processes synchronously identify the respective image frames obtained from the first shared memory.

[0090] Further, for the management device, the set period is determined according to the recognition duration of recognizing one image frame; and / or the first control information is used to instruct the front-end device to store one image frame collected by the front-end device in the first shared memory.

[0091] Further, for the management device, the processor 402 is specifically configured to identify the front-end device that sends the first response message according to the received first response message; obtain the image frame from the first shared memory according to the identification for recognition.

[0092] An embodiment of the present invention further provides a computing device, which may specifically be a desktop computer, a portable computer, a smart phone, a tablet computer, a personal digital assistant (Personal Digital Assistant, PDA), etc. The computing device may include a central processing unit (Center Processing Unit, CPU), a memory, an input / output device, etc. The input device may include a keyboard, a mouse, a touch screen, etc., and the output device may include a display device, such as a liquid crystal display (Liquid Crystal Display, LCD), a cathode ray tube (Cathode Ray Tube, CRT), etc.

[0093] The memory may include a read-only memory (ROM) and a random access memory (RAM), and provide program instructions and data stored in the memory to the processor. In the embodiment of the present invention, the memory may be used to store the program instructions of the image recognition method;

[0094] A processor for calling program instructions stored in the memory and executing an image recognition method according to the obtained program.

[0095] An embodiment of the present invention also provides a computer-readable storage medium storing computer-executable instructions for causing a computer to execute an image recognition method.

[0096] Those skilled in the art should understand that the embodiments of the present invention may be provided as a method, a computer program product. Therefore, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memory, CD-ROM, optical memory, etc.) containing computer-usable program code.

[0097] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present invention. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for implementing the functions specified in one Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0098] These computer program instructions can also be stored in a computer-readable memory capable of guiding a computer or other programmable data processing devices to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured product including an instruction device, and the instruction device implements the functions specified in one Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0099] These computer program instructions can also be loaded onto a computer or other programmable data processing devices, such that a series of operation steps are executed on the computer or other programmable devices to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable devices provide steps for implementing the functions specified in one Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0100] Although the preferred embodiments of the present invention have been described, additional changes and modifications can be made to these embodiments by those skilled in the art once they learn the basic creative concept. Therefore, the appended claims are intended to be construed to include the preferred embodiments as well as all changes and modifications that fall within the scope of the present invention.

[0101] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalent technologies, the present invention is also intended to include these modifications and variations.

Claims

1. An image recognition method, characterized in that, it is applied to a smart home butler provided with multiple shared memories; wherein, for any one of the shared memories, the shared memory is divided into multiple preset positions respectively having a mapping relationship with multiple smart home devices; the method includes: setting N processes, each process is used to recognize the image frames collected by the corresponding smart home device; after the set period arrives, respectively send first control information to N smart home devices through websocket instructions; the first control information is used to instruct the smart home devices to store the collected image frames in the corresponding preset positions in the first shared memory, and to instruct the smart home devices to send a first response message through websocket instructions after storing the image frames in the first shared memory; identifying the smart home device that sends the first response message according to the received first response message; after the next set period arrives, the processes corresponding to the smart home devices with the identifier synchronously recognize the respective image frames obtained from the corresponding preset positions in the first shared memory by using the TensorRT technology, and remove the identifier of the corresponding smart home device; sending second control information to the j-th smart home device through websocket instructions; wherein, the j-th smart home device is the smart home device among the N smart home devices that fails to send the first response message; the second control information is used to instruct the j-th smart home device to retransmit the un-stored image frames collected to the corresponding preset positions in the second shared memory; sending second control information to the k-th smart home device through websocket instructions; wherein, the k-th smart home device is the smart home device among the N smart home devices that successfully sends the first response message; the second control information is used to instruct the k-th smart home device to store the subsequent image frames of the image frames already stored in the first shared memory in the corresponding preset positions in the second shared memory; the set period is determined according to the recognition duration of recognizing at least one adjacent image frame.

2. A smart home butler, characterized in that, it is configured with multiple shared memories; for any one of the shared memories, the shared memory is divided into multiple preset positions respectively having a mapping relationship with multiple smart home devices; the smart home butler further includes: a transceiver, configured to send first control information to N smart home devices through websocket instructions after the set period arrives; the first control information is used to instruct the smart home devices to store the collected image frames in the corresponding preset positions in the first shared memory, and to instruct the smart home devices to send a first response message through websocket instructions after storing the image frames in the first shared memory; A processor, configured to set N processes, each process for identifying an image frame collected by a corresponding smart home device; identify the smart home device that sends the first response message according to the received first response message; after the next set period arrives, the processes corresponding to the smart home devices with the identifier synchronously identify the respective image frames obtained from the corresponding preset positions in the first shared memory using TensorRT technology, and remove the identifier of the corresponding smart home device; The transceiver is further configured to: (1) send second control information to the j-th smart home device through a websocket instruction; wherein, the j-th smart home device is the smart home device among the N smart home devices that has not successfully sent the first response message; the second control information is used to instruct the j-th smart home device to retransmit the un-stored image frames collected to the corresponding preset position in the second shared memory; (2) send second control information to the k-th smart home device through a websocket instruction; wherein, the k-th smart home device is the smart home device among the N smart home devices that has successfully sent the first response message; the second control information is used to instruct the k-th smart home device to store the subsequent image frames of the image frames already stored in the first shared memory to the corresponding preset position in the second shared memory; The set period is determined according to the recognition duration of identifying at least one adjacent image frame.

3. A computing device, characterized in that, comprising: a memory for storing program instructions; a processor for calling the program instructions stored in the memory and executing the method according to claim 1 according to the obtained program.

Images