The invention discloses a man-
machine cooperation
data set construction and
anomaly detection method fusing a multi-
modal large model, and relates to the technical field of man-
machine cooperation, and the method comprises the steps: controlling a virtual
digital human and a virtual mechanical arm to carry out virtual man-
machine cooperation in a virtual man-machine cooperation scene based on a man-machine motion
label, in the process of virtual man-machine cooperation,
human body collision parts, multi-angle man-machine scene images and joint posture data are obtained,
timestamp alignment is further carried out, a multi-
modal data set is formed, the multi-
modal data set is converted into a vision-language
instruction data set through a multi-modal
large model, and the vision-language
instruction data set is used as a virtual man-machine
cooperation model. And utilizing the vision-language
instruction data set to carry out
fine tuning on the multi-modal
large model, and utilizing the multi-modal large model after
fine tuning to realize man-machine cooperation
anomaly detection in the virtual man-machine cooperation scene. According to the method, a high-quality multi-
modal data set can be efficiently constructed,
fine tuning training can be carried out on a multi-modal large model, and man-machine cooperation
anomaly detection in a virtual man-machine cooperation scene is realized.