Equipment control method and device, equipment and medium

Abstract

The invention provides a device control method and apparatus, a device and a medium. The method comprises the steps of obtaining a device image corresponding to a target device; annotating the equipment image to obtain a text pair corresponding to the equipment image; performing function identification according to the equipment image and the text pair to obtain equipment function information corresponding to the equipment image; performing feature extraction according to the equipment image and the equipment function information to obtain a multi-modal equipment function feature corresponding to the target equipment; the equipment function multi-modal knowledge graph corresponding to the multi-modal equipment function features is constructed, the equipment function multi-modal knowledge graph is used for carrying out equipment control on the target equipment, the multi-modal knowledge graph related to the equipment functions is constructed on the basis of the multi-modal information, the equipment functions are accurately recognized through the multi-modal knowledge graph, and the equipment functions are accurately recognized through the multi-modal knowledge graph. Therefore, the equipment is controlled, the function identification of the equipment is realized by using the multi-modal retrieval, and the accuracy of the function control of the equipment is improved.

Description

Technical Field

[0001] The present application relates to the field of intelligent control technology, and in particular to a device control method, a device control apparatus, an electronic device, and a computer-readable storage medium. Background Art

[0002] With the rapid development of the Internet of Things and artificial intelligence technology, smart home has become an important part of modern life. The smart home system integrates various smart devices to provide users with a more convenient and comfortable living environment. The recognition and understanding of device functions are the core of smart home, which affects the intelligence level of smart home system and user experience.

[0003] Although existing algorithm technologies have made great progress in target detection, general scene understanding, and cross-modal retrieval, they have not been well applied in scenarios related to device function identification and control. There is a lack of benchmarks for fine-grained device functions, making it difficult to effectively identify and understand the complex relationships between device functions, resulting in low control accuracy and efficiency of smart home devices. Summary of the invention

[0004] The embodiments of the present application provide a device control method, apparatus, device and medium to overcome the above problems or at least partially solve the above problems.

[0005] The present application discloses a device control method, the method comprising:

[0006] Get the device image corresponding to the target device;

[0007] Annotating the device image to obtain a text pair corresponding to the device image;

[0008] Performing function recognition based on the device image and the text pair to obtain device function information corresponding to the device image;

[0009] Extracting features based on the device image and the device function information to obtain multimodal device function features corresponding to the target device;

[0010] A device function multimodal knowledge graph corresponding to the functional features of the multimodal device is constructed, and the device function multimodal knowledge graph is used to control the target device.

[0011] Optionally, annotating the device image to obtain a text pair corresponding to the device image includes:

[0012] The device image is annotated according to the semantic content of the device image to obtain a text pair corresponding to the device image and a bounding box corresponding to the text pair.

[0013] Optionally, after obtaining the text pair corresponding to the device image and the bounding box corresponding to the text pair, the method further includes:

[0014] In response to a plurality of bounding boxes having the same text pair being located in the same area, a fusion operation is performed on the plurality of bounding boxes; the fusion operation satisfies that a ratio of the areas of the plurality of bounding boxes before and after fusion is greater than a preset threshold.

[0015] Optionally, performing function identification based on the device image and the text to obtain device function information corresponding to the device image includes:

[0016] Recognize the device image according to the target detection model and the text pair to obtain a device function area of ​​the device image;

[0017] The device function area is identified according to the target detection model and the text, and device function information corresponding to the device function area is obtained.

[0018] Optionally, the identifying the device function area according to the target detection model and the text to obtain device function information corresponding to the device function area includes:

[0019] Performing bounding box regression processing on the device functional area to obtain a target device functional area;

[0020] The target device function area is identified according to the target detection model and the text, and device function information corresponding to the target device function area is obtained.

[0021] Optionally, the extracting features according to the device image and the device function information to obtain multimodal device function features corresponding to the target device includes:

[0022] Extracting features from the device image using an image encoder to obtain device function image features of the device image;

[0023] Extracting features from the device function information according to a hierarchical text encoder to obtain device function text features corresponding to the device function information;

[0024] The device image function feature is fused with the device function text feature to obtain the multimodal device function feature.

[0025] Optionally, the fusing the device image function feature with the device function text feature to obtain the multimodal device function feature includes:

[0026] Calculating the device function image feature and the device function text feature according to the contrast loss function to obtain a matching score of the device function image feature and the device function text feature; the matching score is used to determine a fusion weight of the device function image feature and the device function text feature;

[0027] The device function image feature and the device function text feature are fused according to an adaptive pooling strategy and the fusion weight to obtain the multimodal device function feature.

[0028] Optionally, the method further includes:

[0029] The device function multimodal knowledge graph is evaluated according to preset performance evaluation indicators to obtain a performance evaluation result of the device function multimodal knowledge graph.

[0030] Optionally, the device image includes at least multiple types including a device function control image, a device function image and a user image.

[0031] The present application also provides a device control apparatus, the device comprising:

[0032] A device image acquisition module is used to obtain a device image corresponding to a target device;

[0033] A device image annotation module, used to annotate the device image and obtain a text pair corresponding to the device image;

[0034] A device function identification module, used to perform function identification based on the device image and the text pair to obtain device function information corresponding to the device image;

[0035] A multimodal device function feature generation module, used to extract features according to the device image and the device function information to obtain multimodal device function features corresponding to the target device;

[0036] A knowledge graph construction module is used to construct a device function multimodal knowledge graph corresponding to the functional characteristics of the multimodal device, and the device function multimodal knowledge graph is used to control the target device.

[0037] Optionally, the device image annotation module is specifically configured to annotate the device image according to semantic content of the device image to obtain a text pair corresponding to the device image and a bounding box corresponding to the text pair.

[0038] Optionally, the device further comprises:

[0039] The bounding box fusion module is used to perform a fusion operation on multiple bounding boxes in response to multiple bounding boxes with the same text pairs being located in the same area; the fusion operation satisfies that the area ratio of the multiple bounding boxes before and after fusion is greater than a preset threshold.

[0040] Optionally, the device function identification module includes:

[0041] A region recognition submodule, used for recognizing the device image according to the target detection model and the text pair, and obtaining a device function region of the device image;

[0042] The device function identification submodule is used to identify the device function area according to the target detection model and the text, and obtain the device function information corresponding to the device function area.

[0043] Optionally, the device function identification module further includes:

[0044] A bounding box regression processing submodule is used to perform bounding box regression processing on the device functional area to obtain a target device functional area;

[0045] The device function identification submodule is specifically used to identify the target device function area according to the target detection model and the text, and obtain the device function information corresponding to the target device function area.

[0046] Optionally, the multimodal device function feature generation module includes:

[0047] A device function image feature extraction submodule, used for performing feature extraction on the device image according to the image encoder to obtain device function image features of the device image;

[0048] A device function text feature extraction submodule is used to extract features from the device function information according to a hierarchical text encoder to obtain device function text features corresponding to the device function information;

[0049] The multimodal device function feature fusion submodule is used to fuse the device image function feature with the device function text feature to obtain the multimodal device function feature.

[0050] Optionally, the multimodal device function feature fusion submodule includes:

[0051] A matching score calculation unit, used for calculating the device function image feature and the device function text feature according to a contrast loss function to obtain a matching score of the device function image feature and the device function text feature; the matching score is used to determine a fusion weight of the device function image feature and the device function text feature;

[0052] A feature fusion unit is used to fuse the device function image feature and the device function text feature according to an adaptive pooling strategy and the fusion weight to obtain the multimodal device function feature.

[0053] Optionally, the device further comprises:

[0054] The performance evaluation module is used to evaluate the device function multimodal knowledge graph according to preset performance evaluation indicators to obtain a performance evaluation result of the device function multimodal knowledge graph.

[0055] Optionally, the device image includes at least multiple types including a device function control image, a device function image and a user image.

[0056] The embodiment of the present application also discloses an electronic device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus;

[0057] The memory is used to store computer programs;

[0058] The processor is used to implement the method described in the embodiment of the present application when executing the program stored in the memory.

[0059] The embodiment of the present application also discloses a computer-readable storage medium having instructions stored thereon, which, when executed by one or more processors, enables the processors to execute the method described in the embodiment of the present application.

[0060] Compared with the prior art, the embodiments of the present application have the following advantages:

[0061] In an embodiment of the present application, a device image corresponding to a target device is obtained; the device image is annotated to obtain a text pair corresponding to the device image; function recognition is performed based on the device image and text pair to obtain device function information corresponding to the device image; feature extraction is performed based on the device image and device function information to obtain multimodal device function features corresponding to the target device; a device function multimodal knowledge graph corresponding to the multimodal device function features is constructed, and the device function multimodal knowledge graph is used to control the target device. The embodiment of the present application extracts multimodal features of device functions to construct a multimodal knowledge graph of device functions, and then accurately understands the different functions of the device through the multimodal knowledge graph, thereby accurately controlling the device, using multimodal retrieval to identify device functions, improve the accuracy of device function control, and realize intelligent and efficient device control. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0063] Figure 1 is a flowchart of a device control method provided by an embodiment of the present application;

[0064] Figure 2 It is a flow chart of constructing a multimodal knowledge graph of device functions provided in an embodiment of the present application;

[0065] Figure 3 It is a structural block diagram of a device control device provided in an embodiment of the present application;

[0066] Figure 4 is a block diagram of an electronic device provided in an embodiment of the present application;

[0067] Figure 5 It is a schematic diagram of a computer-readable storage medium provided in an embodiment of the present application. DETAILED DESCRIPTION

[0068] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0069] With the rapid development of the Internet of Things and artificial intelligence technology, smart home has become an important part of modern life. The smart home system integrates various smart devices to provide users with a more convenient and comfortable living environment. The recognition and understanding of device functions are the core of smart home, which affects the intelligence level of smart home system and user experience.

[0070] As an example, existing device function control methods usually involve users sending instructions to devices through physical switches, wireless devices, and voice to control the devices to perform corresponding functions, or automatically controlling devices based on user behavior and environmental conditions through machine learning algorithms and artificial intelligence technologies. Although existing algorithm technologies have made great progress in target detection, general scene understanding, and cross-modal retrieval, they have not been well applied in scenarios related to device function identification and control. There is a lack of benchmarks for fine-grained device functions, making it difficult to effectively identify and understand the complex relationships between device functions, resulting in low control accuracy and efficiency of smart home devices.

[0071] In this regard, the advantages of the present application over the prior art are: obtaining a device image corresponding to the target device; annotating the device image to obtain a text pair corresponding to the device image; performing function identification based on the device image and text pair to obtain device function information corresponding to the device image; performing feature extraction based on the device image and device function information to obtain multimodal device function characteristics corresponding to the target device; constructing a multimodal knowledge graph of device functions corresponding to the multimodal device function characteristics, the multimodal knowledge graph of device functions is used to control the target device, and the multimodal features of the device functions are extracted to construct a multimodal knowledge graph of the device functions, and then the different functions of the device are accurately understood through the multimodal knowledge graph, so as to accurately control the device, and the device functions are identified by multimodal retrieval, the accuracy of device function control is improved, and the intelligent and efficient device control is realized.

[0072] Reference Figure 1 , shows a flow chart of the steps of a device control method provided in an embodiment of the present application, which may specifically include the following steps:

[0073] Step 101: Obtain a device image corresponding to a target device;

[0074] In the embodiments of the present application, first, a comprehensive device image is collected as data for device function identification. The device image contains at least one target device. Then, through image recognition technology, different devices and the functions and states of the corresponding devices can be identified and classified. Through analysis and processing of the device image, device identification, device function identification and device control scenario understanding can be achieved. Comprehensive collection of device images provides a basis for subsequent device function understanding and identification.

[0075] In a preferred embodiment of the present application, the device image includes at least multiple types of device function control images, device function images and user images.

[0076] In the embodiment of the present application, by collecting multiple types of device images, it is possible to combine different types of device images and extract more comprehensive features about the device functions, which plays a key role in the subsequent identification and understanding of the device functions. Specifically, the device images include at least three types: device function control images, device function images, and user images. The device function control images refer to images that include devices and paired controls with device function names, providing the most concise description, such as an image that includes an air conditioner and the air conditioner panel displays cooling 26°C; the device function image refers to a detailed text description of the device and the functions included in the device, such as an image that includes an air conditioner and the air conditioner panel displays cooling, dehumidification, energy-saving mode, up and down swing wind and other functions; the user image refers to a device image taken by a user and a text description of the user, such as a user posted a photo of an air conditioner and a fan and described "It's cool to turn on the air conditioner and fan together on this day", and the diversification of device function data is achieved through multiple types of device images.

[0077] Step 102: annotate the device image to obtain a text pair corresponding to the device image;

[0078] In an embodiment of the present application, the device image is annotated according to the content of the device image, and a text pair, i.e., a text description, corresponding to the device image is obtained. Then, the device image and the corresponding device function can be more comprehensively understood through the text pair corresponding to the device image. The text pair can provide a data basis for the subsequent construction of a multimodal knowledge graph, thereby enhancing the understanding of the device image through text pairs, thereby improving the accuracy of device function recognition.

[0079] In a preferred embodiment of the present application, step 102, annotating the device image to obtain a text pair corresponding to the device image, includes:

[0080] The device image is annotated according to the semantic content of the device image to obtain a text pair corresponding to the device image and a bounding box corresponding to the text pair.

[0081] In an embodiment of the present application, a device image is recognized to obtain the semantic content of the device image, and then the device image is annotated according to the semantic content to generate text corresponding to the device image, and the area corresponding to the text is annotated in the device image to generate a corresponding bounding box to obtain a text pair of the device image. For example, if the device image contains a fan and a wind speed increase button and a wind speed decrease button on the fan corresponding panel, a bounding box is generated to cover the wind speed increase button and is annotated as "lower temperature", and a bounding box is generated to cover the wind speed decrease button and is annotated as "increase temperature". It can be understood that the device image includes at least one corresponding text pair, and the text pair represents the semantic information of the device image. In an embodiment of the present application, the device image is annotated according to the semantic content to obtain a corresponding text pair, and then the device image can be understood more comprehensively and intuitively through the text pair corresponding to the device image, so as to improve the accuracy of subsequent identification of the device image corresponding to the device function.

[0082] In a preferred embodiment of the present application, after obtaining the text pair corresponding to the device image and the bounding box corresponding to the text pair, the method further includes:

[0083] In response to a plurality of bounding boxes having the same text pair being located in the same area, a fusion operation is performed on the plurality of bounding boxes; the fusion operation satisfies that a ratio of the areas of the plurality of bounding boxes before and after fusion is greater than a preset threshold.

[0084] In the embodiment of the present application, in the process of annotating the device image, the lack of restrictions on the size and number of bounding boxes may lead to obvious differences in the size of the bounding boxes and the problem of redundancy of the bounding boxes. Therefore, after the annotation of the device image is completed, a label cleaning operation will be performed. Specifically, the label cleaning includes a fusion bounding box operation to determine whether the bounding boxes with the same text pair are located in the same area. If the overlap of multiple bounding boxes is high or the distance is close, they are considered to be located in the same area. Multiple bounding boxes with the same text pair and in an overlapping, mutually contained or adjacent state, that is, located in the same area, are merged into one bounding box through a fusion operation to solve the problem of redundancy of the bounding boxes. Before performing the fusion operation, the area ratio of the bounding boxes before and after the fusion must be calculated to ensure that the area ratio of the bounding boxes before and after the fusion is greater than the preset threshold. If the area ratio is less than the preset threshold, the fusion operation is not performed to ensure the effectiveness and accuracy of the fusion operation and avoid information loss or errors caused by the fusion operation. Among them, the preset threshold can be set to 0.6 according to expert experience. The setting of the preset threshold can be determined by expert experience or statistical analysis, and this application does not limit this.

[0085] Optionally, the annotation cleaning operation may also include filtering bounding boxes with smaller areas and bounding boxes with lower frequencies, thereby reducing redundant information, more accurately describing the function and status of the device, improving the accuracy of multimodal feature extraction, optimizing knowledge graph construction, and achieving precise control of the device.

[0086] As an example, filtering bounding boxes with smaller areas may include the following operations: deleting text pairs and corresponding bounding boxes whose total bounding box area is less than a preset ratio of the area of ​​the entire device image, specifically calculating the area of ​​each bounding box, determining whether the area of ​​the bounding box is less than 3% of the area of ​​the entire device image, and deleting bounding boxes and their corresponding text pairs whose areas are less than the threshold. And if the device image includes at least three bounding boxes with the same text pair, retaining the bounding box with the largest bounding box area under the same text pair and the bounding box with an area greater than a preset ratio of the largest area bounding box, specifically sorting the bounding boxes under the same text pair from large to small according to area, retaining the bounding box with the largest area, determining whether the areas of other bounding boxes are greater than 80% of the largest area bounding box, and retaining the bounding boxes with an area greater than the threshold. Among them, the area ratio can be adjusted as needed, and this application does not impose any restrictions on this.

[0087] As another example, bounding boxes with low appearance frequencies are filtered out in the total device image, specifically, the number of times each text pair appears in the total device image is counted, the appearance frequency of each text pair is calculated, and the text pairs and corresponding bounding boxes with appearance frequencies less than a preset threshold are deleted.

[0088] Step 103: Perform function recognition based on the device image and the text to obtain device function information corresponding to the device image;

[0089] In the embodiment of the present application, the device function information corresponding to the device image is obtained by performing function recognition on the device image and text pair. The text pair provides a semantic description of the device image, which has the effect of enhancing the understanding of the device image. The text pair can help understand the relationship between the device function semantics and the device function, and then identify the corresponding device function information in the device image. Through text pair matching and device function recognition, the embodiment of the present application can more comprehensively understand the device function corresponding to the device image, improve the accuracy of device image recognition, and provide data support for subsequent device function recognition and control.

[0090] In a preferred embodiment of the present application, step 103, performing function identification based on the device image and the text pair to obtain device function information corresponding to the device image, includes:

[0091] Recognize the device image according to the target detection model and the text pair to obtain a device function area of ​​the device image;

[0092] The device function area is identified according to the target detection model and the text, and device function information corresponding to the device function area is obtained.

[0093] In an embodiment of the present application, a device image is identified through a target detection model to obtain an image area in the device image involving device functions, namely, a device function area. The device function corresponding to the device function area is further identified through the target detection model to obtain the device function corresponding to the device image. In this process, text pairs can help the target detection model better understand the semantic information of the device image, and then accurately identify the device function corresponding to the device image and reduce errors.

[0094] Optionally, the target detection model can be a target detection model such as Faster R-CNN (Faster Region-based Convolutional Neural Network), YOLO (You Only Look Once) or DINO (Data-efficient Image Neural Network, unsupervised visual representation learning). Specifically, the test set can be used to evaluate different target detection models, obtain the average precision (AP) of each target detection model on the test set, calculate the average precision of different target detection models under different IoU (Intersection over Union, the overlap between the predicted bounding box and the true bounding box) thresholds to evaluate the performance of the target detection model, and then select the optimal target detection model to further improve the accuracy of device function recognition and achieve more intelligent device control.

[0095] As an example, the Faster R-CNN target detection model is selected. Faster R-CNN first extracts the feature map of the device image based on CNN (Convolutional Neural Network), then generates the device function area based on RPN (Region Proposal Network), and further identifies the device function corresponding to the device function area in combination with the text pair.

[0096] Optionally, in an embodiment of the present application, before the labeled device image data set is input into the target detection model for device function identification, the bounding boxes can also be sorted from large to small according to the bounding box area, and the three largest areas of the bounding box are retained as true values. By retaining the most important target areas, the training efficiency and detection performance of the target detection model are improved, and more intelligent device control is achieved.

[0097] Optionally, in an embodiment of the present application, before the labeled device image data set is input into the target detection model for device function identification, the text pairs can also be converted according to the preset device function hierarchy to text pairs corresponding to the previous level, such as "temperature increase" and "temperature decrease" are both merged into "temperature category", "wind speed increase" and "wind speed decrease" are both merged into "wind speed category", which can avoid the problem that the target detection model cannot identify highly similar categories in the same branch, reduce the complexity of the device image data set, and improve the generalization ability of the target detection model, so that it can better adapt to different device functions and achieve more intelligent device control. Among them, the device function hierarchy can set a suitable device function hierarchy based on previous experience and experimental data, or calculate the distribution of the device function hierarchy to set the device function hierarchy, and the present application does not limit this.

[0098] In a preferred embodiment of the present application, the identifying the device function area according to the target detection model and the text pair to obtain the device function information corresponding to the device function area includes:

[0099] Performing bounding box regression processing on the device functional area to obtain a target device functional area;

[0100] The target device function area is identified according to the target detection model and the text, and device function information corresponding to the target device function area is obtained.

[0101] In the embodiment of the present application, before identifying the device function corresponding to the device function area, the position and size of the bounding box corresponding to the device function area are adjusted by a boundary line regression algorithm so that it more accurately covers the device function area in the device image, thereby improving the accuracy of device function recognition. Optionally, the predicted bounding box of the target detection model can be constrained by a bounding box regression loss function to make it closer to the real bounding box, thereby improving the detection performance of the model.

[0102] Step 104: extracting features based on the device image and the device function information to obtain multimodal device function features corresponding to the target device;

[0103] In an embodiment of the present application, feature extraction is performed on a device image to obtain features related to device functions in the device image, namely, device function image features. At the same time, feature extraction is performed on the device function information to obtain text features related to device functions in the device function information, namely, device function text features. The device function image features and the device function text features are then spliced ​​and fused by dimension to obtain multimodal device function features. The multimodal device function features can be used to more comprehensively represent the device function information, thereby providing a data basis for constructing a multimodal knowledge graph.

[0104] In a preferred embodiment of the present application, the step of extracting features based on the device image and the device function information to obtain multimodal device function features corresponding to the target device includes:

[0105] Extracting features from the device image using an image encoder to obtain device function image features of the device image;

[0106] Extracting features from the device function information according to a hierarchical text encoder to obtain device function text features corresponding to the device function information;

[0107] The device image function feature is fused with the device function text feature to obtain the multimodal device function feature.

[0108] In an embodiment of the present application, two independent feature extractors are used to extract device function image features and device function text features respectively, and then the embedding distance between the matched device function image features and device function text features is reduced in an end-to-end manner through comparison constraints. Finally, the device function image features and the device function text features are fused to obtain multimodal device function features.

[0109] Specifically, according to the following formula (1), the input device image is feature extracted through the image encoder, and then the features of the input device image are projected into a common embedding space using a linear fully connected layer to obtain the functional features of the device image:

[0110] f M =||F c (E M (M))||2; (1)

[0111] Among them, f M represents the functional characteristics of the device image, M is the input device image, E M represents the image encoder, F c represents a linear fully connected layer, ||F c (E M (M))||2 indicates the c (E M (M) Perform l2 normalization.

[0112] According to the following formula (2), the input device function information is feature extracted through the hierarchical text encoder, and then the features of the input device function information are projected into the common embedding space using the linear fully connected layer to obtain the device function text features:

[0113] f G =||F c(T2(T1(ing1),...,t1(ING o )))||2; (2)

[0114] Among them, f G G is the device function text feature, i =[ing1,ing2…,ing o ] is the input device function information, T1 and T2 layered Transformer encoder, F c Represents a linear fully connected layer.

[0115] In a preferred embodiment of the present application, the fusing of the device image function feature with the device function text feature to obtain the multimodal device function feature includes:

[0116] Calculating the device function image feature and the device function text feature according to the contrast loss function to obtain a matching score of the device function image feature and the device function text feature; the matching score is used to determine a fusion weight of the device function image feature and the device function text feature;

[0117] The device function image feature and the device function text feature are fused according to an adaptive pooling strategy and the fusion weight to obtain the multimodal device function feature.

[0118] In an embodiment of the present application, the cosine similarity between the device function image features and the device function text features is calculated by contrasting the loss function, the cosine similarity is used as the matching score of the device function image features and the device function text features, the fusion weight of the device function image features and the device function text features is determined according to the matching score, and further, the device function image features and the device function text features are weightedly fused according to the fusion weight according to the adaptive pooling strategy to obtain multimodal device function features.

[0119] Specifically, the cosine similarity is calculated according to the contrast loss function to determine the matching score according to the following formula (3) and formula (4):

[0120]

[0121] Among them, f M Represents the functional characteristics of the device image, with dimensions N, f G is the device function text feature, the dimension is M, T is the preset threshold, sim i,i Indicates the similarity of the positive sample pair, sim i,j , represents the similarity of negative sample pairs.

[0122] The adaptive pooling strategy is further combined to automatically fuse the device function image features and the device function text features according to the matching scores to obtain multimodal device function features.

[0123] Step 105: construct a device function multimodal knowledge graph corresponding to the functional characteristics of the multimodal device, and the device function multimodal knowledge graph is used to control the target device.

[0124] In an embodiment of the present application, the extracted multimodal device function features are integrated into a knowledge graph to form a structured knowledge base for describing the various functions and operations of the device, and then a query is performed through the device function multimodal knowledge graph, and then the device control operation is automatically performed according to the query results in the knowledge graph. For example, when a user inputs device images including air conditioners and televisions, the device function multimodal knowledge graph obtains corresponding device function results such as lowering the air conditioner temperature and turning on the TV by identifying the image features and text features of the device images, and then automatically performs the corresponding control operations, and utilizes multimodal retrieval to realize the recognition of graphic device function control, improve the accuracy and efficiency of device control, and realize intelligent and efficient device control.

[0125] In a preferred embodiment of the present application, the method further includes:

[0126] The device function multimodal knowledge graph is evaluated according to preset performance evaluation indicators to obtain a performance evaluation result of the device function multimodal knowledge graph.

[0127] In an embodiment of the present application, the device function multimodal knowledge graph is evaluated according to preset performance evaluation indicators to obtain a performance evaluation result of the device function multimodal knowledge graph. Through the performance evaluation, the performance of the device function multimodal knowledge graph can be fully understood to further optimize the device function multimodal knowledge graph.

[0128] Optionally, when using the device function multimodal knowledge graph to perform device function identification retrieval, the median index and the correct sample ratio can be used as performance evaluation indicators. The median index is the median position of the correct samples in the query results. The median index reflects the overall ranking quality of the retrieval results. The smaller the median index, the higher the position of the correct sample in the retrieval results, and the higher the quality of the retrieval results. The correct sample ratio is the percentage of the correct sample index in the previous sample retrieved. The correct sample ratio reflects the accuracy of the retrieval results. The higher the correct sample ratio, the higher the accuracy of the retrieval results. The embodiment of the present application can comprehensively evaluate the retrieval result quality of the device function multimodal knowledge graph through the median index and the correct sample ratio, and then optimize the device function multimodal knowledge graph based on the evaluation results.

[0129] In an embodiment of the present application, a device image corresponding to a target device is obtained; the device image is annotated to obtain a text pair corresponding to the device image; function recognition is performed based on the device image and text pair to obtain device function information corresponding to the device image; feature extraction is performed based on the device image and device function information to obtain multimodal device function features corresponding to the target device; a device function multimodal knowledge graph corresponding to the multimodal device function features is constructed, and the device function multimodal knowledge graph is used to control the target device. The embodiment of the present application extracts multimodal features of device functions to construct a multimodal knowledge graph of device functions, and then accurately understands the different functions of the device through the multimodal knowledge graph, thereby accurately controlling the device, using multimodal retrieval to identify device functions, improve the accuracy of device function control, and realize intelligent and efficient device control.

[0130] In one example, Figure 2 As shown, a flowchart for constructing a multimodal knowledge graph of device functions provided in an embodiment of the present application is shown. In the embodiment of the present application, a device image of the target device, i.e., an air-conditioning image, is first obtained, and a text pair, i.e., air-conditioning temperature reduction, is obtained by annotating the device image according to the semantics; then, the text features in the device image are identified through a target detection algorithm, such as device (air-conditioning), function (temperature, reduction), manufacturer (air-conditioning brand A, air-conditioning brand B), and the corresponding image features in the device image are identified through a target detection algorithm; finally, based on the combination of image features and text features of different modalities of the device image, the image features-text features are associated to supplement the multimodal relationship to construct a complete multimodal knowledge graph of device functions.

[0131] It should be noted that, for the method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present application are not limited by the described order of actions, because according to the embodiments of the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present application.

[0132] Reference Figure 3 , shows a structural block diagram of a device control device provided in an embodiment of the present application, which may specifically include the following modules:

[0133] The device image acquisition module 301 is used to obtain the device image corresponding to the target device;

[0134] The device image annotation module 302 is used to annotate the device image to obtain a text pair corresponding to the device image;

[0135] A device function identification module 303 is used to perform function identification based on the device image and the text pair to obtain device function information corresponding to the device image;

[0136] A multimodal device function feature generation module 304 is used to extract features according to the device image and the device function information to obtain multimodal device function features corresponding to the target device;

[0137] The knowledge graph construction module 305 is used to construct a device function multimodal knowledge graph corresponding to the functional characteristics of the multimodal device, and the device function multimodal knowledge graph is used to control the target device.

[0138] In some embodiments of the present application, the device image annotation module 302 is specifically used to annotate the device image according to the semantic content of the device image to obtain a text pair corresponding to the device image and a bounding box corresponding to the text pair.

[0139] In some embodiments of the present application, the device further includes:

[0140] The bounding box fusion module is used to perform a fusion operation on multiple bounding boxes in response to multiple bounding boxes with the same text pairs being located in the same area; the fusion operation satisfies that the area ratio of the multiple bounding boxes before and after fusion is greater than a preset threshold.

[0141] In some embodiments of the present application, the device function identification module 303 includes:

[0142] A region recognition submodule, used for recognizing the device image according to the target detection model and the text pair, and obtaining a device function region of the device image;

[0143] The device function identification submodule is used to identify the device function area according to the target detection model and the text, and obtain the device function information corresponding to the device function area.

[0144] In some embodiments of the present application, the device function identification module 303 further includes:

[0145] A bounding box regression processing submodule is used to perform bounding box regression processing on the device functional area to obtain a target device functional area;

[0146] The device function identification submodule is specifically used to identify the target device function area according to the target detection model and the text, and obtain the device function information corresponding to the target device function area.

[0147] In some embodiments of the present application, the multimodal device function feature generation module 304 includes:

[0148] A device function image feature extraction submodule, used for performing feature extraction on the device image according to the image encoder to obtain device function image features of the device image;

[0149] A device function text feature extraction submodule is used to extract features from the device function information according to a hierarchical text encoder to obtain device function text features corresponding to the device function information;

[0150] The multimodal device function feature fusion submodule is used to fuse the device image function feature with the device function text feature to obtain the multimodal device function feature.

[0151] In some embodiments of the present application, the multimodal device function feature fusion submodule includes:

[0152] A matching score calculation unit, used for calculating the device function image feature and the device function text feature according to a contrast loss function to obtain a matching score of the device function image feature and the device function text feature; the matching score is used to determine a fusion weight of the device function image feature and the device function text feature;

[0153] A feature fusion unit is used to fuse the device function image feature and the device function text feature according to an adaptive pooling strategy and the fusion weight to obtain the multimodal device function feature.

[0154] In some embodiments of the present application, the device further includes:

[0155] The performance evaluation module is used to evaluate the device function multimodal knowledge graph according to preset performance evaluation indicators to obtain a performance evaluation result of the device function multimodal knowledge graph.

[0156] In some embodiments of the present application, the device image includes at least multiple types of device function control images, device function images, and user images.

[0157] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0158] In addition, the present application also provides an electronic device, such as Figure 4 As shown, it includes a processor 401, a communication interface 402, a memory 403 and a communication bus 404, wherein the processor 401, the communication interface 402, and the memory 403 communicate with each other through the communication bus 404.

[0159] Memory 403, used for storing computer programs;

[0160] The processor 401 is used to implement the method described in the above embodiment when executing the program stored in the memory 403.

[0161] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0162] The communication interface is used for communication between the above terminal and other devices.

[0163] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0164] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0165] like Figure 5 As shown, in another embodiment provided in the present application, a computer-readable storage medium 501 is also provided, in which instructions are stored. When executed by one or more processors, the processors execute the methods described in the above embodiments.

[0166] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0167] Those skilled in the art will appreciate that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0168] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0169] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0170] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0171] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0172] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0173] The above is a detailed introduction to a device control method, apparatus, equipment and medium provided. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in the field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A device control method, characterized in that: The method comprises: Get the device image corresponding to the target device; Annotating the device image to obtain a text pair corresponding to the device image; Performing function recognition based on the device image and the text pair to obtain device function information corresponding to the device image; Extracting features based on the device image and the device function information to obtain multimodal device function features corresponding to the target device; A device function multimodal knowledge graph corresponding to the functional features of the multimodal device is constructed, and the device function multimodal knowledge graph is used to control the target device.

2. The method according to claim 1, characterized in that: The annotating the device image to obtain a text pair corresponding to the device image includes: The device image is annotated according to the semantic content of the device image to obtain a text pair corresponding to the device image and a bounding box corresponding to the text pair.

3. The method according to claim 2, characterized in that: After obtaining the text pair corresponding to the device image and the bounding box corresponding to the text pair, the method further includes: In response to a plurality of bounding boxes having the same text pair being located in the same area, a fusion operation is performed on the plurality of bounding boxes; the fusion operation satisfies that a ratio of the areas of the plurality of bounding boxes before and after fusion is greater than a preset threshold.

4. The method according to claim 1, characterized in that: The performing function identification based on the device image and the text to obtain device function information corresponding to the device image includes: Recognize the device image according to the target detection model and the text pair to obtain a device function area of ​​the device image; The device function area is identified according to the target detection model and the text, and device function information corresponding to the device function area is obtained.

5. The method according to claim 4, characterized in that: The identifying the device function area according to the target detection model and the text to obtain device function information corresponding to the device function area includes: Performing bounding box regression processing on the device functional area to obtain a target device functional area; The target device function area is identified according to the target detection model and the text, and device function information corresponding to the target device function area is obtained.

6. The method according to claim 1, characterized in that: The extracting features according to the device image and the device function information to obtain multimodal device function features corresponding to the target device includes: Extracting features from the device image using an image encoder to obtain device function image features of the device image; Extracting features from the device function information according to a hierarchical text encoder to obtain device function text features corresponding to the device function information; The device image function feature is fused with the device function text feature to obtain the multimodal device function feature.

7. The method according to claim 6, characterized in that: The fusing the device image function feature with the device function text feature to obtain the multimodal device function feature includes: Calculating the device function image feature and the device function text feature according to the contrast loss function to obtain a matching score of the device function image feature and the device function text feature; the matching score is used to determine a fusion weight of the device function image feature and the device function text feature; The device function image feature and the device function text feature are fused according to an adaptive pooling strategy and the fusion weight to obtain the multimodal device function feature.

8. The method according to claim 1, characterized in that: The method further comprises: The device function multimodal knowledge graph is evaluated according to preset performance evaluation indicators to obtain a performance evaluation result of the device function multimodal knowledge graph.

9. The method according to claim 1, characterized in that: The device images include at least a device function control image, a device function image and a user image.

10. A device control device, characterized in that: The device comprises: A device image acquisition module is used to obtain a device image corresponding to a target device; A device image annotation module, used to annotate the device image and obtain a text pair corresponding to the device image; A device function identification module, used to perform function identification based on the device image and the text pair to obtain device function information corresponding to the device image; A multimodal device function feature generation module, used to extract features according to the device image and the device function information to obtain multimodal device function features corresponding to the target device; A knowledge graph construction module is used to construct a device function multimodal knowledge graph corresponding to the functional characteristics of the multimodal device, and the device function multimodal knowledge graph is used to control the target device.

11. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; The memory is used to store computer programs; The processor is used to implement the method according to any one of claims 1 to 9 when executing the program stored in the memory.

12. A computer-readable storage medium having instructions stored thereon, which, when executed by one or more processors, cause the processors to perform the method according to any one of claims 1 to 9.

Images