Equipment control method and system, electronic equipment and storage medium

Abstract

The invention provides an equipment control method and system, electronic equipment and a storage medium. The method comprises the following steps: acquiring a real control panel image of a real control panel corresponding to the equipment; performing image recognition on the real control panel image to obtain information of each real element in the real control panel image, generating a virtual control panel based on the information of all the real elements, and performing interaction function configuration on the virtual control panel to control the equipment through the virtual control panel; the virtual control panel can be automatically generated when remote control of the equipment is realized, manual intervention is reduced, the development period is shortened, the development cost is reduced, and different types of equipment can be automatically adapted, so that complex and diversified equipment requirements can be quickly met, and the user experience is improved.

Description

Technical Field

[0001] This application relates to the technical field of device control, and particularly to a device control method, system, electronic device, and storage medium. Background Art

[0002] In fields such as industry, household appliances, and medical equipment, it is usually the user who controls the device locally or on-site by operating a physical control panel, which has certain limitations. With the rapid development of technology, the user's demand for remote control of devices is also increasing day by day. Especially in scenarios such as industrial automation, smart home, and remote medical treatment, remote control can not only improve the convenience of operation, but also significantly enhance work efficiency and user experience.

[0003] Currently, the implementation of most remote controls still relies on manual development of control panels, and this method has many problems: First, the development cycle is long and the efficiency is low, making it difficult to quickly respond to changes in market demand; Second, in the face of complex and diverse device requirements, developers need to conduct separate designs and developments for each device, resulting in a large amount of development work and high development costs. Summary of the Invention

[0004] In view of the above-mentioned disadvantages of the prior art, this application provides a device control method, system, electronic device, and storage medium to solve the technical problems that the implementation of remote control mainly relies on manual development of control panels, with a long development cycle, low efficiency, and a large amount of development work and high development costs in the face of complex and diverse device requirements.

[0005] This application provides a device control method, the method includes: obtaining a real control panel image of the device corresponding to the real control panel; performing image recognition on the real control panel image to obtain information of each real component in the real control panel image; based on the information of all real components, generating a virtual control panel and configuring interaction functions for the virtual control panel to control the device through the virtual control panel.

[0006] In an embodiment of this application, performing image recognition on the real control panel image to obtain information of each real component in the real control panel image includes: performing target detection on the real control panel image to obtain the position, size, and category of each real component; based on the position of each real component, performing shape recognition on each real component in the real control panel image to obtain the shape of each real component; based on the position of each real component, performing text recognition on the real control panel image to obtain the text content of each real component; taking the position, size, category, shape, and text content of each real component as the information of each real component.

[0007] In an embodiment of the present application, based on the information of all real components, a virtual control panel is generated and the interactive functions of the virtual control panel are configured, including: drawing a virtual component according to the size and shape of a real component, and establishing a corresponding relationship between the real component and the virtual component to obtain a plurality of virtual components; arranging the virtual components according to the positions of the real components to generate the virtual control panel; determining the functions of the virtual components and the interaction logic between the virtual components according to the categories and text contents of the real components, so as to configure the interactive functions of the virtual control panel based on the functions of the virtual components and the interaction logic between the virtual components.

[0008] In an embodiment of the present application, the device is controlled through the virtual control panel, including: establishing a connection between the device and the front end, where the virtual control panel is configured on the front end; generating a control instruction based on the operation information of the user on the virtual control panel, and triggering the front end to send the control instruction to the device to control the device according to the control instruction.

[0009] In an embodiment of the present application, establishing the connection between the device and the front end includes: the device establishes a long connection with the message middleware and sends a device registration message; the cloud listens through the message middleware, and after listening to the device going online, pushes a device online message to the front end to establish the connection between the device and the front end.

[0010] In an embodiment of the present application, the front end sending the control instruction to the device includes: the front end sending the control instruction to the cloud; the cloud publishing the control instruction to the message middleware, and sending the control instruction to the device through the message middleware.

[0011] In an embodiment of the present application, obtaining the real control panel image of the device corresponding to the real control panel includes: obtaining a plurality of initial images of the real control panel of the device corresponding to the device; performing histogram processing on each initial image to obtain the brightness histogram corresponding to each initial image, and comparing the preset brightness threshold with the brightness of each pixel in the brightness histogram corresponding to each initial image, and screening out a plurality of candidate images from the plurality of initial images according to the comparison result; calculating the gray variance and gradient value of each candidate image to obtain the gray variance and gradient value of each candidate image, and determining the evaluation value of each candidate image based on the gray variance and gradient value of each candidate image, and using the candidate image with the highest evaluation value as the real control panel image.

[0012] In an embodiment of the present application, a device control system is further provided. The system includes: an image acquisition module for acquiring a real control panel image of the device corresponding to the real control panel; an information processing module for performing image recognition on the real control panel image to obtain information of each real component in the real control panel image; generating a virtual control panel based on the information of all real components and performing interaction function configuration on the virtual control panel; and a device control module for controlling the device through the virtual control panel.

[0013] In an embodiment of the present application, an electronic device is further provided. The electronic device includes: one or more processors; a storage device for storing one or more programs, which when executed by the one or more processors, cause the electronic device to implement the device control method as described above.

[0014] In an embodiment of the present application, a computer-readable storage medium is further provided, on which a computer program is stored. When the computer program is executed by a processor of a computer, the computer is caused to execute the device control method as described above.

[0015] Advantages of the present invention: The present invention provides a device control method, system, electronic device and storage medium. The method extracts information of real components from a real control panel image through image recognition, generates a virtual control panel based on the information of real components and completes interaction function configuration, so as to control the device through the virtual control panel. When realizing remote control of the device, it can automatically generate a virtual control panel, reduce manual intervention, shorten the development cycle and reduce the development cost. And it can automatically adapt to different types of devices, so as to quickly respond to complex and diverse device requirements and improve the user experience. In addition, the method can also quickly respond to the update of device functions, only need to re-acquire the real control panel image and generate a new virtual control panel, without re-development, greatly reducing the maintenance cost.

[0016] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a schematic diagram of an implementation environment of a device control method shown in an exemplary embodiment of the present application;

[0018] Figure 2 is a flowchart of a device control method shown in an exemplary embodiment of the present application;

[0019] Figure 3 is a system architecture diagram of remote control shown in a specific embodiment of the present application;

[0020] Figure 4 It is a block diagram of a device control system shown in an exemplary embodiment of the present application;

[0021] Figure 5 It is a schematic structural diagram of an electronic device shown in an exemplary embodiment of the present application. Specific Embodiments

[0022] The following uses specific specific examples to illustrate the implementation manners of the present application. Those skilled in the art can easily understand other advantages and effects of the present application from the content disclosed in this specification. The present application can also be implemented or applied through other different specific implementation manners. Various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that, without conflict, the following embodiments and the features in the embodiments can be combined with each other.

[0023] It should be noted that the diagrams provided in the following embodiments only illustrate the basic concept of the present application in a schematic manner. Therefore, only the components related to the present application are shown in the diagrams, rather than being drawn according to the number, shape, and size of the components in actual implementation. The types, quantities, and proportions of the components in actual implementation can be arbitrarily changed, and the component layout type may also be more complex.

[0024] It should be noted that in the present application, "first", "second", etc. are only used to distinguish similar objects, and are not used to limit the order or sequence of similar objects. The described "including", "having", etc. are deformed, indicating that the scope covered by the subject of the word does not exclude other examples except the examples shown by the word.

[0025] It can be understood that the various numerical numbers, step numbers, etc. recorded in the present application are for the convenience of description and are not used to limit the scope of the present application. The size of the reference numerals in the present application does not mean the sequence of execution order, and the execution order of each process should be determined by its function and internal logic.

[0026] In the following description, a large number of details are discussed to provide a more thorough explanation of the embodiments of the present application. However, it is obvious to those skilled in the art that the embodiments of the present application can be implemented without these specific details. In other embodiments, well-known structures and devices are shown in the form of block diagrams rather than in detail to avoid making the embodiments of the present application difficult to understand.

[0027] Embodiments of the present application respectively propose a device control method, a device control system, an electronic device, a computer-readable storage medium, and a computer program product. The following will describe these embodiments in detail.

[0028] Please refer to Figure 1 , Figure 1 which is a schematic diagram of the implementation environment of a device control method shown in an exemplary embodiment of the present application.

[0029] As Figure 1 shown, the implementation environment may include an image acquisition device 110 and a computer device 120. Among them, the image acquisition device 110 may be at least one of a camera, a webcam, a scanner, etc., and the computer device 120 may be at least one of a GPU (Graphic Processing Unit) computer, a GPU computing cluster, a neural network computer, etc., and no limitations are imposed here. The real control panel of the device can be photographed or scanned using the image acquisition device 110, and the real control panel image is collected and provided to the computer device 120. The computer device 120 processes the real control panel image to generate a virtual control panel to achieve the control of the device. Of course, the user can also obtain the real control panel image through a network or a database and input it into the computer device 120 for processing, and no limitations are imposed here either.

[0030] Schematically, the computer device 120 acquires the real control panel image of the device corresponding to the real control panel; performs image recognition on the real control panel image to obtain information on each real component in the real control panel image; based on the information of all real components, generates a virtual control panel and configures the interaction function of the virtual control panel to control the device through the virtual control panel. It can be seen that the technical solution of the embodiment of the present application can realize the remote control of the device by automatically generating a virtual control panel, reduce manual intervention, shorten the development cycle, and reduce the development cost. And it can automatically adapt to different types of devices, so as to quickly respond to complex and diverse device requirements and improve the user experience. In addition, this technical solution can also quickly respond to the update of the device function, and only needs to re-acquire the real control panel image and generate a new virtual control panel, without re-development, which greatly reduces the maintenance cost.

[0031] It should be noted that the device control method provided by the embodiment of the present application is generally specifically executed by the computer device 120, and the corresponding device control system is generally set in the computer device 120.

[0032] Please refer to Figure 2 , Figure 2 which is a flowchart of a device control method shown in an exemplary embodiment of the present application. This device control method can be applied to Figure 1The described implementation environment, and is specifically executed by the computer device 120 in this implementation environment. It should be understood that this device control method can also be applied to other exemplary implementation environments and be specifically executed by devices in other implementation environments. This embodiment does not limit the implementation environment to which this device control method applies. As Figure 2 As shown, in an exemplary embodiment, this device control method at least includes steps S210 to S230, which are introduced in detail as follows:

[0033] Step S210, obtain a real control panel image of the device corresponding to the real control panel.

[0034] In an embodiment of the present application, the device can be a household device such as an air conditioner or a TV, or a device in an industrial production line, or a device in a laboratory. The weight of the device is not limited here. The real control panel corresponding to the device refers to the physical control panel used to control the device, such as an air conditioner remote control, an air conditioner panel, a TV remote control, etc. The user can use a high-definition imaging device such as a camera, a camera, or a scanner to take pictures of the real control panel corresponding to the device to collect an image of the real control panel as the real control panel image. For a complex panel, a multi-view stitching technology can also be used to generate a complete real control panel image. Of course, the user can also search for an image of the real control panel corresponding to the device through network big data as the real control panel image. The acquisition method of the real control panel image is not limited here.

[0035] To improve the accuracy of subsequent image recognition, it can be required that the real control panel image is clear and non-reflective. Pay attention to the light and shooting skills during shooting to ensure that a high-quality real control panel image is captured. Multiple real control panel images can also be obtained, and high-quality real control panel images can be selected from multiple real control panel images.

[0036] In an embodiment of the present application, step S210 includes: obtaining multiple initial images of the real control panel corresponding to the device; performing histogram processing on each initial image to obtain the brightness histogram corresponding to each initial image, and comparing the preset brightness threshold with the brightness of each pixel in the brightness histogram corresponding to each initial image, and screening out multiple candidate images from multiple initial images according to the comparison result; calculating the gray variance and gradient value of each candidate image to obtain the gray variance and gradient value of each candidate image, and determining the evaluation value of each candidate image based on the gray variance and gradient value of each candidate image, and using the candidate image with the highest evaluation value as the real control panel image.

[0037] In this embodiment, the real control panel of the device can be photographed multiple times to obtain multiple images of the real control panel, which are used as multiple initial images. Then, each initial image is converted into a grayscale image, and the luminance histogram of each grayscale image is calculated as the luminance histogram corresponding to each initial image. For each luminance histogram, a preset luminance threshold is compared with the luminance of each pixel in the luminance histogram. According to the comparison result, it is determined whether there is a highlight area in the luminance histogram. If not, the initial image corresponding to the luminance histogram is used as a candidate image. In this way, multiple candidate images are obtained. Among them, there are many ways to determine whether there is a highlight area in the luminance histogram according to the comparison result. For example, if the ratio of the number of pixels with luminance exceeding the preset luminance threshold in the luminance histogram to the total number of pixels in the luminance histogram exceeds a preset ratio, it is determined that there is a highlight area in the luminance histogram; otherwise, there is no highlight area. Another example is that if the pixels with luminance exceeding the preset luminance threshold in the luminance histogram are concentratedly distributed in the luminance histogram, it is determined that there is a highlight area in the luminance histogram; otherwise, there is no highlight area.

[0038] After obtaining multiple candidate images, for each candidate image, preprocessing such as denoising and grayscale conversion is performed on the candidate image. The variance of the grayscale values in the preprocessed candidate image is calculated as the grayscale variance of the candidate image, and the sum of the squares of the grayscale differences between adjacent pixels in the preprocessed candidate image, including horizontal and vertical directions, is calculated to obtain the energy gradient value as the gradient value of the candidate image. The grayscale variance and gradient value of the candidate image are normalized, and one of summation, average value calculation, or weighted average value calculation is performed on the normalized values. The calculation result is used as the evaluation value of the candidate image. In this way, the evaluation values of multiple candidate images are obtained. The evaluation values of each candidate image are sorted. The higher the evaluation value, the higher the clarity of the candidate image. Therefore, the candidate image with the highest evaluation value can be used as the real control panel image.

[0039] Through the above two screenings, a clear and non-reflective real control panel image is obtained.

[0040] Step S220: Perform image recognition on the real control panel image to obtain information about each real component in the real control panel image.

[0041] In an embodiment of the present application, a real component refers to a real component in the real control panel that is used for a user to perform operations to control the device or to display device information, such as buttons, switches, knobs, displays, indicator lights, etc. Deep learning models such as YOLO and Faster R-CNN can be used to detect real components in the real control panel image, including buttons, switches, knobs, and displays, etc., and obtain information about each real component, including at least one of position, size, category, shape, and text content.

[0042] In one embodiment of the present application, step S220 includes: performing object detection on the real control panel image to obtain the positions, sizes, and categories of each real component; performing shape recognition on each real component in the real control panel image based on the positions of each real component to obtain the shapes of each real component; performing text recognition on the real control panel image based on the positions of each real component to obtain the text content of each real component; and using the positions, sizes, categories, shapes, and text content of each real component as the information of each real component.

[0043] In this embodiment, object detection can be performed on each real component in the real control panel image through an object detection model to obtain the coordinate positions, sizes, and categories of each real component in the real control panel image. Among them, the categories of real components include buttons, switches, knobs, displays, etc. The object detection model is pre-trained through a variety of different types of real control panel sample images.

[0044] Shape recognition can be performed on each real component based on the coordinate positions of each real component. Schematically, the local image of each real component can be segmented from the real control panel image according to the coordinate positions of each real component, and the contour information of the real component can be extracted from the local image of the real component through a contour detection algorithm, and geometric features, including area, perimeter, bounding box, aspect ratio, circularity, polygon approximation, etc., can be calculated based on the contour information. Based on the geometric features and preset rules, the shape of the real component can be determined. For example: if the aspect ratio is close to 1 and the circularity is high, it is determined to be circular; if the number of vertices is 4 and the aspect ratio is close to 1, it is determined to be square; if the number of vertices is 4 and the aspect ratio is far from 1, it is determined to be rectangular; if the number of vertices is greater than 4, it is determined to be a polygon. Among them, a pre-trained machine learning model such as a support vector machine, random forest, or neural network can be used to classify the geometric features, or a convolutional neural network can be directly used to classify the local image of the real component to output the shape category.

[0045] Text recognition can also be performed on the real control panel image based on the coordinate positions of each real component to obtain the text content of each real component. Schematically, the local image of each real component can be segmented from the real control panel image according to the coordinate positions of each real component, and then OCR (Optical Character Recognition) technology can be used to extract the text information from the local image of the real component as the text content of the real component.

[0046] The positions, sizes, categories, shapes, and text content of the real components are used together as the information of the real components.

[0047] In this embodiment, through multi-modal recognition such as object detection, shape recognition, and text recognition, the recognition accuracy of real components can be improved.

[0048] Step S230: Based on the information of all real components, generate a virtual control panel and configure the interactive functions of the virtual control panel to control the device through the virtual control panel.

[0049] In an embodiment of the present application, based on the information of all recognized real components, map the real components to device operation functions, determine the functions that each real component is responsible for triggering or controlling in device operation. After determining the mapping relationship between the real components and the operation functions, construct a functional logic model to describe how these functions are interrelated and cooperate with each other. Then, based on the functional logic model, develop a virtual control panel through web front-end technology to generate an interactive virtual control panel.

[0050] In an embodiment of the present application, generating a virtual control panel and configuring the interactive functions of the virtual control panel based on the information of all real components includes: drawing a virtual component according to the size and shape of a real component, and establishing a correspondence between the real component and the virtual component to obtain multiple virtual components; arranging the virtual components according to the positions of the real components to generate a virtual control panel; determining the functions of the virtual components and the interaction logic between the virtual components according to the categories and text contents of the real components, so as to configure the interactive functions of the virtual control panel based on the functions of the virtual components and the interaction logic between the virtual components.

[0051] In this embodiment, a virtual component refers to the virtual component corresponding to a real component in a real control panel in a virtual control panel, such as a virtual switch, a virtual button, a virtual display screen, etc. SVG or Canvas technology can be used to draw virtual components, and component-based development can be implemented through front-end frameworks such as Vue.js or React.js. For example: Vue.js technology can be used to draw Ant Design Vue components according to the size and shape of real components as virtual components, and the virtual components can be laid out based on the coordinate positions of real components to form a preliminary virtual control panel. Among them, SVG can be used to draw the shape of components or Canvas to draw dynamic graphics. At the same time, according to the categories and text contents of each real component, the functional logic of the device is mapped, including: defining the functions of components (referring to real components and virtual components corresponding to real components) such as start and stop, and analyzing the interaction logic between components such as the linkage between buttons and display screens to build a functional logic model. Based on the functional logic model, konva.js can be used to implement the interaction functions of the virtual control panel, such as button clicks and knob sliding. Specifically, the functional logic model can be parsed to generate a JSON Schema, and the virtual components formed by rendering AntDesign Vue components based on the JSON Schema can be used. In interaction optimization, for knob operations, visualization rotation can be achieved by combining a-slider and konva.js; for status feedback, a-notification can be used to display the operation results in real time; for historical records, a-table can be integrated to display operation logs. By drawing Ant Design Vue components to form virtual components, the development efficiency of the virtual control panel can be improved.

[0052] In an embodiment of the present application, the device is controlled through a virtual control panel, including: establishing a connection between the device and the front end, where the virtual control panel is configured on the front end; generating a control instruction based on the operation information of the user on the virtual control panel, and triggering the front end to send the control instruction to the device to control the device according to the control instruction.

[0053] In this embodiment, the front end refers to a control terminal device used to control a device through a virtual control panel, such as a mobile phone, a tablet, a computer, etc. Therefore, a responsive design can also be used to adapt the virtual control panel to the screen sizes of different control terminal devices. The connection between the device and the front end can be established through the network address information of the device, where the network address information includes a Bluetooth address, a MAC address, an IP address, etc., and the connection method between the device and the front end can be a Bluetooth connection, a local area network connection, an Internet connection, etc. After the connection is established, the user can operate the virtual components in the virtual control panel through the front end to generate control instructions, so that the front end sends the control instructions to the device through the established connection channel to achieve short-range or remote control of the device.

[0054] In an embodiment of the present application, establishing a connection between a device and a front end includes: the device establishing a long connection with a message middleware and sending a device registration message; the cloud listening through the message middleware, and after detecting that the device goes online, pushing a device online message to the front end to establish a connection between the device and the front end.

[0055] In an embodiment of the present application, the front end sending a control instruction to the device includes: the front end sending the control instruction to the cloud; the cloud publishing the control instruction to the message middleware, and sending the control instruction to the device through the message middleware.

[0056] In this embodiment, the connection between the device and the front end can be established through a network communication module such as the WebSocket protocol, and real-time two-way synchronization and data interaction between the virtual control panel and the device are achieved through network communication to ensure the timely sending and feedback of control instructions.

[0057] Please refer to Figure 3 , Figure 3 which is the system architecture diagram of remote control shown in a specific embodiment of the present application. As Figure 3 shown, the system mainly includes a device end (i.e., the device), a Web front end (i.e., the front end), a Mosquitto Broker, and a KeepServer. In the device connection establishment stage, after the device end is powered on, it establishes a long connection with the Mosquitto Broker, which is used as a message middleware, through the MQTT protocol (port 1883) and sends a device registration message; the KeepServer, as the cloud, listens to the $SYS / devices / + / connect topic, captures the device online event, records the device online status in Redis, notifies the Node.js service to update the device list through gRPC, and pushes a device status change notification to the Web front end to prompt the device to go online, thereby establishing a connection between the device and the front end.

[0058] In the control instruction issuing stage, the user operates virtual components such as sliding on the Ant Design Vue through the Web front end <a-slider>, generate control instructions; the Web front-end sends the control instructions to the KeepServer through WebSocket. The KeepServer performs protocol conversion, including verifying the validity of the JWT token, converting the control instructions in the HTTP / WebSocket message format into the MQTT format, and publishing them to the device-specific topic devices / D001 / control, so that the MosquittoBroker sends the control instructions in the MQTT format to the device side through the MQTT protocol.

[0059] In addition, the system can also implement device status reporting, front-end status synchronization, and exception handling.

[0060] In the device status reporting stage, the device side can publish status data to the devices / D001 / status topic at regular intervals (such as 1 second) through the sensor; the KeepServer subscribes to all devices / + / status topics, performs data format standardization processing such as unit conversion or time alignment, filters out-of-range values based on preset thresholds, and persists and stores them in the IOTDB time series database.

[0061] In the front-end status synchronization stage, the Node.js service pushes status updates to the Web front-end through Server-Sent Events (SSE) to achieve dynamic updates of Ant Design Vue components.

[0062] In the exception handling stage, for the handling of network interruptions, when the device side enables the MQTT Last Will message to notify abnormal offline, the KeepServer can trigger the disconnection reconnection mechanism using the exponential backoff algorithm and display an <a-alert type="error"> warning prompt on the Web front-end; for the handling of failed instruction transmission, when the KeepServer detects that MQTT QoS=1 does not receive an ACK confirmation, it can store the failed control instructions in the Redis retry queue, retry sending every 60 seconds, up to 3 times, and finally generate <a-notification>Error log.

[0063] The system can also enable TLS1.3 encryption throughout the transport layer and perform instruction signature and access control at the application layer to provide secure communication guarantee for data interaction between the device and the front end. Specifically, X.509 certificate authentication can be performed at the device end, two-way HTTPS verification can be performed at the Web front end, the instruction signature can be generated by using the HMAC-SHA256 algorithm to generate a message digest, and the access control can be based on the device operation permission management of the RBAC model.

[0064] The system can improve the message arrival rate through KeepServer to ensure the reliability of communication. Moreover, the system supports dual-protocol access of MQTT / HTTP, enhancing the compatibility of the system.

[0065] Please refer to Figure 4 , Figure 4 is a block diagram of a device control system shown in an exemplary embodiment of the present application. The system can be applied to Figure 1 the implementation environment shown, and is specifically configured in the computer device 120. The system can also be applicable to other exemplary implementation environments and is specifically configured in other devices. This embodiment does not limit the implementation environment applicable to the system. As Figure 4 shown, the exemplary device control system includes: an image acquisition module 410, configured to acquire a real control panel image of the device corresponding to the real control panel; an information processing module 420, configured to perform image recognition on the real control panel image to obtain information of each real component in the real control panel image; generate a virtual control panel based on the information of all real components and perform interaction function configuration on the virtual control panel; a device control module 430, configured to control the device through the virtual control panel.

[0066] In this embodiment, the image acquisition module 410 may be a camera, a webcam, a scanner, or an electronic device such as a mobile phone, a tablet, or a computer with a camera. The information processing module 420 may be a computer, a computing cluster, a microcomputer, an embedded computer, a neural network computer, a processor, a server, the cloud, etc. The device control module 430 may be an electronic device with a touch panel such as a mobile phone, a tablet, or a computer, and no limitation is imposed here. The image acquisition module 410, the information processing module 420, and the device control module 430 may be integrated into the computer device 120. For example, the image acquisition module 410 may be the camera of the computer device 120, the information processing module 420 may be the processor of the computer device 120, and the device control module 430 may include the display, the processor, and the network interface card of the computer device 120. Of course, the image acquisition module 410, the information processing module 420, and the device control module 430 may also be independent computer devices, and no limitation is imposed here either.

[0067] The device control system provided by the embodiments of the present application realizes the near - range or remote control of a device through image acquisition, image recognition, panel generation, and device control. It can automatically generate a virtual control panel, reduce the manual development time, support the mapping of multiple device types without customized development, and can be widely used in various scenarios, including the remote control of household air conditioners, the monitoring of industrial production line equipment, and the monitoring of intelligent laboratory equipment. For the remote control of a household air conditioner, the user can use a mobile phone as the image acquisition module 410 to take a picture of the air conditioner panel and upload it to the cloud server. The cloud server, as the information processing module 420, recognizes the buttons and knobs on the panel, generates a virtual control panel, and sends it to the mobile phone. The mobile phone, as the device control module 430, realizes remote temperature adjustment and mode switching. For the monitoring of industrial production line equipment, the operator can use a mobile phone as the image acquisition module 410 to take a picture of the production line equipment control panel and send it to the computer terminal. The computer terminal, as the information processing module 420, automatically generates a virtual control panel and, as the device control module 430, realizes remote start, stop, and parameter adjustment, etc. For the monitoring of intelligent laboratory equipment, the experimenter can use a mobile phone as the image acquisition module 410 to take a picture of the laboratory equipment control panel and send it to the computer terminal. The computer terminal, as the information processing module 420, automatically generates a virtual control panel and, as the device control module 430, realizes remote monitoring or adjustment, etc.

[0068] Exemplarily, in the above three remote control scenarios, the device control module 430 can control the device by using MQTT over WebSocket. After the temperature control device knob is recognized, a corresponding virtual component can be formed by generating an a-slider component. The pressure gauge reading can be displayed through an a-progress circular progress bar. An abnormal state can trigger an a-alert for warning. Historical data can generate a trend chart through an a-charts.

[0069] It should be noted that the device control system provided in the above embodiment and the device control method provided in the above embodiment belong to the same concept. The specific ways in which each module and unit perform operations have been described in detail in the method embodiment, and will not be elaborated here. In practical applications, the device control system provided in the above embodiment can, according to needs, allocate the above functions to different functional modules, that is, divide the internal structure of the system into different functional modules to complete all or part of the functions described above. This is not limited here either.

[0070] This embodiment also provides an electronic device, including: one or more processors; a storage device for storing one or more programs, which when executed by the one or more processors, enable the electronic device to implement the device control methods provided in the above various embodiments.

[0071] Please refer to Figure 5 , Figure 5 which is a schematic structural diagram of an electronic device shown in an exemplary embodiment of the present application. It should be noted that Figure 5 the electronic device 500 shown is only an example and should not bring any limitations to the functions and usage scopes of the embodiments of the present application.

[0072] As Figure 5 shown, the electronic device 500 includes a processor 501, a memory 502, and a communication bus 503; the communication bus 503 is used to connect the processor 501 and the memory 502; the processor 501 is used to execute the computer program stored in the memory 502 to implement one or more of the methods as in the above embodiments.

[0073] This embodiment also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor of the computer, the computer is enabled to execute the device control method as described above. The computer-readable storage medium may be included in the electronic device described in the above embodiment, or may exist separately without being assembled into the electronic device.

[0074] This embodiment also provides a computer program product or a computer program. The computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the device control method provided in each of the above embodiments.

[0075] The electronic device provided in this embodiment includes a processor, a memory, a transceiver, and a communication interface. The memory and the communication interface are connected to the processor and the transceiver and complete communication with each other. The memory is used to store a computer program, the communication interface is used for communication, and the processor and the transceiver are used to run the computer program to enable the electronic device to execute each step of the above method.

[0076] In this embodiment, the memory may include a random access memory (Random Access Memory, abbreviated as RAM), and may also include a non-volatile memory, such as at least one disk memory.

[0077] The above-mentioned processor may be a general-purpose processor, including a central processing unit (Central Processing Unit, abbreviated as CPU), a network processor (Network Processor, abbreviated as NP), etc.; it may also be a digital signal processor (Digital Signal Processing, abbreviated as DSP), an application specific integrated circuit (Application Specific Integrated Circuit, abbreviated as ASIC), a field programmable gate array (Field-Programmable Gate Array, abbreviated as FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0078] For the computer-readable storage medium in this embodiment, those of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiments can be completed by hardware related to the computer program. The foregoing computer program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps including the above method embodiments; and the foregoing storage medium includes: ROM (read-only memory), RAM (random access memory), magnetic disk or optical disk and other media that can store program codes.

[0079] The above embodiments are only illustrative of the principles and effects of the present application and are not intended to limit the present application. Any person familiar with this technology can modify or change the above embodiments without departing from the spirit and scope of the present application. Therefore, all equivalent modifications or changes completed by those with ordinary knowledge in the technical field without departing from the spirit and technical idea disclosed by the present application should still be covered by the claims of the present application.

Claims

1. A device control method, characterized in that: The method comprises: Obtain a real control panel image corresponding to the real control panel of the device; Performing image recognition on the real control panel image to obtain information of each real component in the real control panel image; Based on the information of all real components, a virtual control panel is generated and interactive functions of the virtual control panel are configured so as to control the device through the virtual control panel.

2. The device control method according to claim 1, characterized in that: Performing image recognition on the real control panel image to obtain information of each real component in the real control panel image includes: Performing target detection on the real control panel image to obtain the position, size and category of each real component; Based on the position of each real component, shape recognition is performed on each real component in the real control panel image to obtain the shape of each real component; Based on the position of each real component, performing text recognition on the real control panel image to obtain the text content of each real component; The position, size, category, shape and text content of each real component are used as the information of each real component.

3. The device control method according to claim 2, characterized in that: Based on the information of all real components, a virtual control panel is generated and interactive functions of the virtual control panel are configured, including: Drawing a virtual component according to the size and shape of a real component, and establishing a corresponding relationship between the real component and the virtual component to obtain a plurality of virtual components; Arranging the virtual components according to the positions of the real components to generate the virtual control panel; The function of each virtual component and the interaction logic between virtual components are determined according to the category and text content of each real component, so as to configure the interaction function of the virtual control panel based on the function of each virtual component and the interaction logic between virtual components.

4. The device control method according to claim 1, characterized in that: Controlling the device through the virtual control panel includes: Establishing a connection between the device and a front end, wherein the virtual control panel is configured at the front end; A control instruction is generated based on the user's operation information on the virtual control panel, and the front end is triggered to send the control instruction to the device, so as to control the device according to the control instruction.

5. The device control method according to claim 4, characterized in that: Establishing a connection between the device and the front end, including: The device establishes a long connection with the message middleware and sends a device registration message; The cloud monitors through the message middleware, and after monitoring that the device is online, pushes the device online message to the front end to establish a connection between the device and the front end.

6. The device control method according to claim 5, characterized in that: The front end sends the control instruction to the device, including: The front end sends the control instruction to the cloud; The cloud publishes the control instruction to the message middleware, and sends the control instruction to the device through the message middleware.

7. The device control method according to any one of claims 1 to 6, characterized in that: Get the real control panel image of the device corresponding to the real control panel, including: Acquire a plurality of initial images of a real control panel corresponding to the device; Performing histogram processing on each initial image to obtain a brightness histogram corresponding to each initial image, and comparing a preset brightness threshold with the brightness of each pixel in the brightness histogram corresponding to each initial image, and selecting a plurality of candidate images from the plurality of initial images according to the comparison result; Grayscale variance and gradient value calculations are performed on each candidate image to obtain the grayscale variance and gradient value of each candidate image, and an evaluation value of each candidate image is determined based on the grayscale variance and gradient value of each candidate image, and the candidate image with the highest evaluation value is used as the real control panel image.

8. A device control system, characterized in that: The system comprises: An image acquisition module, used to obtain an image of a real control panel corresponding to a real control panel of the device; An information processing module, configured to perform image recognition on the real control panel image to obtain information of each real component in the real control panel image; based on the information of all real components, generate a virtual control panel and configure interactive functions for the virtual control panel; The device control module is used to control the device through the virtual control panel.

9. An electronic device, characterized in that: The electronic device comprises: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the electronic device to implement the device control method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor of a computer, the computer is caused to execute the device control method according to any one of claims 1 to 7.

Images